2021-09-17T01:37:44.080 INFO:root:teuthology version: 1.1.0-63133ca9 2021-09-17T01:37:44.081 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name ivancich-2021-09-16_17:11:22-rgw-wip-eric-testing-2-distro-basic-smithi --owner scheduled_ivancich@teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} --archive /home/teuthworker/archive/ivancich-2021-09-16_17:11:22-rgw-wip-eric-testing-2-distro-basic-smithi/6392884 -- /home/teuthworker/archive/ivancich-2021-09-16_17:11:22-rgw-wip-eric-testing-2-distro-basic-smithi/6392884/orig.config.yaml 2021-09-17T01:37:44.125 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-09-17T01:37:44.181 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/ivancich-2021-09-16_17:11:22-rgw-wip-eric-testing-2-distro-basic-smithi/6392884 branch: wip-eric-testing-2 description: rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} email: ivancich@redhat.com first_in_suite: false job_id: '6392884' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: ivancich-2021-09-16_17:11:22-rgw-wip-eric-testing-2-distro-basic-smithi nuke-on-error: true os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-eric-testing-2 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 2a911e49300877ce783c2becf9f484a85fa1c049 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 2a911e49300877ce783c2becf9f484a85fa1c049 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 rgw-multisite-tests: args: - tests.py - tests_ps.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-eric-testing-2 sha1: 2a911e49300877ce783c2becf9f484a85fa1c049 owner: scheduled_ivancich@teuthology priority: 83 repo: git://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 2a911e49300877ce783c2becf9f484a85fa1c049 sleep_before_teardown: 0 suite: rgw suite_branch: wip-eric-testing-2 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_2a911e49300877ce783c2becf9f484a85fa1c049/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 2a911e49300877ce783c2becf9f484a85fa1c049 targets: smithi013.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQUSlRIEp7IBLYORccR8lUdTmBJ7wmpIUUDFB3BhcT3ksU2F+8QkMCOspCnZF1sYWLW7px9JhbwBIFpTNh82FaIYm+10eqZjBj0nwoTruKjA2DxTEat/JJX2bMF71TbOrjDQwZccmz8TZGYhIlK2tHMoOPB8rGEcM3AHllSFf64/M7lUiZlIZOOEChRvepzkPTfdyHE7lBCH+PJ9pCabnK17OjVgmBGEkUWbbprw6HadBeQ1CrgVzwjabqyAd/8DCvTY2OxE/k+sVVZe4U6kEdBcRe9erCVlQxjoXlPchpL+k0hA2+rk8LHkurz/mdT9av4QVYxi8YcTKIfjYbwhgq5eu964h+rl3y4oZ/NIk+X8MOzsHViYYfHA5WXCRDcZxqUojCCL5+VntRfbe/cwNK9UGdV4LJ3F9dNSw9tYEAvtzE7Mh7xzHOPntmtg/pL43RboxM/YqQEp5wAGNFLPBjhhfsx+54JpWVZBYw5hjU9m36xlETxeS60ZMpA9ZpiuE= smithi045.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCulDN10/Vke3BN23zcEbwKxvhTR910c5AgG82/Ohf4WiNPYA98666PzF0PZudRGkkX89oUG+zPhqFxQc+j3oGHtRyORPLZzJPG//ZKLtwr1WONI012azN1JgotupRG+CwfHM7sThE09XdXiYLDYsIj9R5NDPT8ATTKxRAMVpMUuPx0MjyS9R7c4mVQ+ZKEDq8hRS1hl5svH+iBtEooVtLGZljHsj8HvHDjdtrrTxbSkzhZxHlnD6pEm2SngtbAglAiMO5NuYltRDf8x/k6V3eBODOz4rJUdDtsif9nCjU2HUEcdnNTyZlRJizeshvySk67OoZ5fc2eezTmluQ6QIcv+B0dLUG3Eor1oqHCdtxzBR/PJMZo9tLppdtH9AHitodCgOkdMJyd/zzLBF9iDLBsrXPQ20wVwb5rqnqzRZIpz28zreE9wrS07Ny5FfkexvGofoqbbDpjm8sshrtwv/jfcPcTYd0NUhL49SuSyhOWFGpYKvx8pgAmEwz2EFTvfVc= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master teuthology_sha1: 63133ca92c299bd23f40f94d370cf5381b0b1313 timestamp: 2021-09-16_17:11:22 tube: smithi user: ivancich verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 2021-09-17T01:37:44.181 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_2a911e49300877ce783c2becf9f484a85fa1c049/qa; will attempt to use it 2021-09-17T01:37:44.182 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_2a911e49300877ce783c2becf9f484a85fa1c049/qa/tasks 2021-09-17T01:37:44.182 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-09-17T01:37:44.184 INFO:teuthology.task.internal:Checking packages... 2021-09-17T01:37:44.206 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '2a911e49300877ce783c2becf9f484a85fa1c049' 2021-09-17T01:37:44.207 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-09-17T01:37:44.207 INFO:teuthology.packaging:ref: None 2021-09-17T01:37:44.208 INFO:teuthology.packaging:tag: None 2021-09-17T01:37:44.208 INFO:teuthology.packaging:branch: wip-eric-testing-2 2021-09-17T01:37:44.208 INFO:teuthology.packaging:sha1: 2a911e49300877ce783c2becf9f484a85fa1c049 2021-09-17T01:37:44.209 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-eric-testing-2 2021-09-17T01:37:44.379 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-7730.g2a911e49 2021-09-17T01:37:44.380 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-09-17T01:37:44.391 INFO:teuthology.task.internal:no buildpackages task found 2021-09-17T01:37:44.391 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-09-17T01:37:44.435 INFO:teuthology.task.internal:Saving configuration 2021-09-17T01:37:44.450 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-09-17T01:37:44.462 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-17T01:37:44.479 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-17 01:33:13.961832', 'description': '/home/teuthworker/archive/ivancich-2021-09-16_17:11:22-rgw-wip-eric-testing-2-distro-basic-smithi/6392884', 'locked_by': 'scheduled_ivancich@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGatuNtjMJD1+PvNiRqz7qCt1iKZZjo6YQLq98MDj6TOGMO2XDOFp21BM4oFhhsnnv9vEli9Xmbh1pWSEbe5J48=', 'os_version': '8.4', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi013.front.sepia.ceph.com'} 2021-09-17T01:37:44.495 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-17 01:33:13.961812', 'description': '/home/teuthworker/archive/ivancich-2021-09-16_17:11:22-rgw-wip-eric-testing-2-distro-basic-smithi/6392884', 'locked_by': 'scheduled_ivancich@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPuPgSJaMav966mDb7F+5bnUv1+h7zuDintex4DNaMC16kQIcSRHnfuxtYz7ORzwatyYl9qTCzlqnDJE7D6wTXs=', 'os_version': '8.4', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi045.front.sepia.ceph.com'} 2021-09-17T01:37:44.495 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-09-17T01:37:44.507 INFO:teuthology.task.internal:roles: ubuntu@smithi013.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-09-17T01:37:44.507 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-09-17T01:37:44.508 INFO:teuthology.run_tasks:Running task console_log... 2021-09-17T01:37:44.577 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efd8b261f28>, signals=[15]) 2021-09-17T01:37:44.578 INFO:teuthology.run_tasks:Running task internal.connect... 2021-09-17T01:37:44.593 INFO:teuthology.task.internal:Opening connections... 2021-09-17T01:37:44.593 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi013.front.sepia.ceph.com 2021-09-17T01:37:44.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi013.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T01:37:44.667 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2021-09-17T01:37:44.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T01:37:44.737 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-09-17T01:37:44.749 DEBUG:teuthology.orchestra.run.smithi013:> uname -m 2021-09-17T01:37:44.765 INFO:teuthology.orchestra.run.smithi013.stdout:x86_64 2021-09-17T01:37:44.766 DEBUG:teuthology.orchestra.run.smithi013:> cat /etc/os-release 2021-09-17T01:37:44.821 INFO:teuthology.orchestra.run.smithi013.stdout:NAME="Red Hat Enterprise Linux" 2021-09-17T01:37:44.821 INFO:teuthology.orchestra.run.smithi013.stdout:VERSION="8.4 (Ootpa)" 2021-09-17T01:37:44.821 INFO:teuthology.orchestra.run.smithi013.stdout:ID="rhel" 2021-09-17T01:37:44.822 INFO:teuthology.orchestra.run.smithi013.stdout:ID_LIKE="fedora" 2021-09-17T01:37:44.822 INFO:teuthology.orchestra.run.smithi013.stdout:VERSION_ID="8.4" 2021-09-17T01:37:44.822 INFO:teuthology.orchestra.run.smithi013.stdout:PLATFORM_ID="platform:el8" 2021-09-17T01:37:44.822 INFO:teuthology.orchestra.run.smithi013.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-09-17T01:37:44.823 INFO:teuthology.orchestra.run.smithi013.stdout:ANSI_COLOR="0;31" 2021-09-17T01:37:44.823 INFO:teuthology.orchestra.run.smithi013.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-09-17T01:37:44.823 INFO:teuthology.orchestra.run.smithi013.stdout:HOME_URL="https://www.redhat.com/" 2021-09-17T01:37:44.823 INFO:teuthology.orchestra.run.smithi013.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-09-17T01:37:44.824 INFO:teuthology.orchestra.run.smithi013.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-09-17T01:37:44.824 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:37:44.824 INFO:teuthology.orchestra.run.smithi013.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-09-17T01:37:44.824 INFO:teuthology.orchestra.run.smithi013.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-09-17T01:37:44.824 INFO:teuthology.orchestra.run.smithi013.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-09-17T01:37:44.825 INFO:teuthology.orchestra.run.smithi013.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-09-17T01:37:44.825 INFO:teuthology.lock.ops:Updating smithi013.front.sepia.ceph.com on lock server 2021-09-17T01:37:44.841 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2021-09-17T01:37:44.858 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2021-09-17T01:37:44.859 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2021-09-17T01:37:44.915 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="Red Hat Enterprise Linux" 2021-09-17T01:37:44.915 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="8.4 (Ootpa)" 2021-09-17T01:37:44.915 INFO:teuthology.orchestra.run.smithi045.stdout:ID="rhel" 2021-09-17T01:37:44.915 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="fedora" 2021-09-17T01:37:44.916 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="8.4" 2021-09-17T01:37:44.916 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el8" 2021-09-17T01:37:44.916 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-09-17T01:37:44.916 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2021-09-17T01:37:44.917 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-09-17T01:37:44.917 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://www.redhat.com/" 2021-09-17T01:37:44.917 INFO:teuthology.orchestra.run.smithi045.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-09-17T01:37:44.917 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-09-17T01:37:44.918 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:37:44.918 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-09-17T01:37:44.918 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-09-17T01:37:44.918 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-09-17T01:37:44.918 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-09-17T01:37:44.919 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2021-09-17T01:37:44.933 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-09-17T01:37:44.948 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-09-17T01:37:44.969 INFO:teuthology.task.internal:Checking for old test directory... 2021-09-17T01:37:44.969 DEBUG:teuthology.orchestra.run.smithi013:> test '!' -e /home/ubuntu/cephtest 2021-09-17T01:37:44.972 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2021-09-17T01:37:44.988 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-09-17T01:37:44.999 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-09-17T01:37:44.999 DEBUG:teuthology.orchestra.run.smithi013:> test -z $(ls -A /var/lib/ceph) 2021-09-17T01:37:45.029 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2021-09-17T01:37:45.047 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-09-17T01:37:45.091 INFO:teuthology.run_tasks:Running task kernel... 2021-09-17T01:37:45.111 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-09-17T01:37:45.111 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-09-17T01:37:45.112 DEBUG:teuthology.orchestra.run.smithi013:> uname -r 2021-09-17T01:37:45.129 INFO:teuthology.orchestra.run.smithi013.stdout:4.18.0-305.el8.x86_64 2021-09-17T01:37:45.130 INFO:teuthology.task.kernel:Running kernel on smithi013: 4.18.0-305.el8.x86_64 2021-09-17T01:37:45.130 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum install -y kernel 2021-09-17T01:37:46.494 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:37:46.494 INFO:teuthology.orchestra.run.smithi013.stdout:Unable to read consumer identity 2021-09-17T01:37:46.597 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:37:46.597 INFO:teuthology.orchestra.run.smithi013.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-17T01:37:46.597 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:37:48.484 INFO:teuthology.orchestra.run.smithi013.stdout:Extra Packages for Enterprise Linux 14 MB/s | 10 MB 00:00 2021-09-17T01:37:50.642 INFO:teuthology.orchestra.run.smithi013.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2021-09-17T01:37:51.182 INFO:teuthology.orchestra.run.smithi013.stdout:Package kernel-4.18.0-305.el8.x86_64 is already installed. 2021-09-17T01:37:51.287 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:37:51.289 INFO:teuthology.orchestra.run.smithi013.stdout:Nothing to do. 2021-09-17T01:37:51.289 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:37:51.421 DEBUG:teuthology.orchestra.run.smithi013:> echo no | sudo yum reinstall kernel || true 2021-09-17T01:37:51.792 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:37:51.792 INFO:teuthology.orchestra.run.smithi013.stdout:Unable to read consumer identity 2021-09-17T01:37:51.833 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:37:51.834 INFO:teuthology.orchestra.run.smithi013.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-17T01:37:51.834 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:37:52.021 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:02 ago on Fri 17 Sep 2021 01:37:50 AM UTC. 2021-09-17T01:37:52.139 INFO:teuthology.orchestra.run.smithi013.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-17T01:37:52.173 INFO:teuthology.orchestra.run.smithi013.stderr:Error: No packages marked for reinstall. 2021-09-17T01:37:52.218 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum reinstall -y kernel || true 2021-09-17T01:37:52.552 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:37:52.552 INFO:teuthology.orchestra.run.smithi013.stdout:Unable to read consumer identity 2021-09-17T01:37:52.591 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:37:52.592 INFO:teuthology.orchestra.run.smithi013.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-17T01:37:52.592 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:37:52.778 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:02 ago on Fri 17 Sep 2021 01:37:50 AM UTC. 2021-09-17T01:37:52.899 INFO:teuthology.orchestra.run.smithi013.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-17T01:37:52.929 INFO:teuthology.orchestra.run.smithi013.stderr:Error: No packages marked for reinstall. 2021-09-17T01:37:52.973 DEBUG:teuthology.orchestra.run.smithi013:> rpm -q kernel --last | head -n 1 2021-09-17T01:37:53.044 INFO:teuthology.orchestra.run.smithi013.stdout:kernel-4.18.0-305.el8.x86_64 Thu 20 May 2021 08:07:00 PM UTC 2021-09-17T01:37:53.046 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.el8.x86_64 2021-09-17T01:37:53.046 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-09-17T01:37:53.046 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2021-09-17T01:37:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:4.18.0-305.el8.x86_64 2021-09-17T01:37:53.065 INFO:teuthology.task.kernel:Running kernel on smithi045: 4.18.0-305.el8.x86_64 2021-09-17T01:37:53.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2021-09-17T01:37:54.463 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:37:54.464 INFO:teuthology.orchestra.run.smithi045.stdout:Unable to read consumer identity 2021-09-17T01:37:54.540 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:37:54.541 INFO:teuthology.orchestra.run.smithi045.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-17T01:37:54.541 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:37:56.652 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 13 MB/s | 10 MB 00:00 2021-09-17T01:37:58.800 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2021-09-17T01:37:59.401 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-4.18.0-305.el8.x86_64 is already installed. 2021-09-17T01:37:59.532 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:37:59.533 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-09-17T01:37:59.533 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:37:59.671 DEBUG:teuthology.orchestra.run.smithi045:> echo no | sudo yum reinstall kernel || true 2021-09-17T01:38:00.006 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:38:00.006 INFO:teuthology.orchestra.run.smithi045.stdout:Unable to read consumer identity 2021-09-17T01:38:00.049 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:38:00.049 INFO:teuthology.orchestra.run.smithi045.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-17T01:38:00.049 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:38:00.239 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:02 ago on Fri 17 Sep 2021 01:37:58 AM UTC. 2021-09-17T01:38:00.358 INFO:teuthology.orchestra.run.smithi045.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-17T01:38:00.401 INFO:teuthology.orchestra.run.smithi045.stderr:Error: No packages marked for reinstall. 2021-09-17T01:38:00.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum reinstall -y kernel || true 2021-09-17T01:38:00.779 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:38:00.780 INFO:teuthology.orchestra.run.smithi045.stdout:Unable to read consumer identity 2021-09-17T01:38:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:38:00.828 INFO:teuthology.orchestra.run.smithi045.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-17T01:38:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:38:01.015 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:03 ago on Fri 17 Sep 2021 01:37:58 AM UTC. 2021-09-17T01:38:01.189 INFO:teuthology.orchestra.run.smithi045.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-17T01:38:01.221 INFO:teuthology.orchestra.run.smithi045.stderr:Error: No packages marked for reinstall. 2021-09-17T01:38:01.271 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel --last | head -n 1 2021-09-17T01:38:01.349 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-4.18.0-305.el8.x86_64 Thu 20 May 2021 08:07:00 PM UTC 2021-09-17T01:38:01.350 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.el8.x86_64 2021-09-17T01:38:01.351 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-09-17T01:38:01.351 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-09-17T01:38:01.351 DEBUG:teuthology.orchestra.run.smithi013:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-17T01:38:01.408 INFO:teuthology.orchestra.run.smithi013.stdout:ttyS1 2021-09-17T01:38:01.420 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-09-17T01:38:01.421 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-17T01:38:01.478 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2021-09-17T01:38:01.493 INFO:teuthology.run_tasks:Running task internal.base... 2021-09-17T01:38:01.503 INFO:teuthology.task.internal:Creating test directory... 2021-09-17T01:38:01.503 DEBUG:teuthology.orchestra.run.smithi013:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-17T01:38:01.506 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-17T01:38:01.522 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-09-17T01:38:01.541 INFO:teuthology.run_tasks:Running task internal.archive... 2021-09-17T01:38:01.557 INFO:teuthology.task.internal:Creating archive directory... 2021-09-17T01:38:01.558 DEBUG:teuthology.orchestra.run.smithi013:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-17T01:38:01.565 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-17T01:38:01.592 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-09-17T01:38:01.599 INFO:teuthology.task.internal:Enabling coredump saving... 2021-09-17T01:38:01.600 DEBUG:teuthology.orchestra.run.smithi013:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-17T01:38:01.626 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-17T01:38:01.659 INFO:teuthology.orchestra.run.smithi013.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-17T01:38:01.671 INFO:teuthology.orchestra.run.smithi013.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-17T01:38:01.673 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-17T01:38:01.684 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-17T01:38:01.685 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-09-17T01:38:01.694 INFO:teuthology.task.internal:Configuring sudo... 2021-09-17T01:38:01.694 DEBUG:teuthology.orchestra.run.smithi013:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-17T01:38:01.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-17T01:38:01.743 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-09-17T01:38:01.755 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-09-17T01:38:01.755 DEBUG:teuthology.orchestra.run.smithi013:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-17T01:38:01.786 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-17T01:38:01.801 DEBUG:teuthology.orchestra.run.smithi013:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-17T01:38:01.883 DEBUG:teuthology.orchestra.run.smithi013:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-17T01:38:01.970 DEBUG:teuthology.orchestra.run.smithi013:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-17T01:38:02.007 DEBUG:teuthology.orchestra.run.smithi013:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-17T01:38:02.060 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:38:02.061 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-17T01:38:02.130 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-17T01:38:02.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-17T01:38:02.271 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-17T01:38:02.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-17T01:38:02.354 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:38:02.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-17T01:38:02.418 DEBUG:teuthology.orchestra.run.smithi013:> sudo service rsyslog restart 2021-09-17T01:38:02.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2021-09-17T01:38:02.466 INFO:teuthology.orchestra.run.smithi013.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-09-17T01:38:02.504 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-09-17T01:38:02.912 INFO:teuthology.run_tasks:Running task internal.timer... 2021-09-17T01:38:03.056 INFO:teuthology.task.internal:Starting timer... 2021-09-17T01:38:03.057 INFO:teuthology.run_tasks:Running task pcp... 2021-09-17T01:38:03.071 INFO:teuthology.run_tasks:Running task selinux... 2021-09-17T01:38:03.104 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-09-17T01:38:03.105 DEBUG:teuthology.orchestra.run.smithi013:> sudo service auditd rotate 2021-09-17T01:38:03.179 INFO:teuthology.orchestra.run.smithi013.stdout:Rotating logs: [ OK ] 2021-09-17T01:38:03.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2021-09-17T01:38:03.244 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: [ OK ] 2021-09-17T01:38:03.246 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-09-17T01:38:03.247 DEBUG:teuthology.orchestra.run.smithi013:> /usr/sbin/getenforce 2021-09-17T01:38:03.270 INFO:teuthology.orchestra.run.smithi013.stdout:Permissive 2021-09-17T01:38:03.271 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2021-09-17T01:38:03.294 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2021-09-17T01:38:03.294 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi013.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi045.front.sepia.ceph.com': 'permissive'} 2021-09-17T01:38:03.295 DEBUG:teuthology.orchestra.run.smithi013:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-09-17T01:38:03.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T01:38:03.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-09-17T01:38:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T01:38:03.363 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-09-17T01:38:03.363 DEBUG:teuthology.orchestra.run.smithi013:> sudo /usr/sbin/setenforce permissive 2021-09-17T01:38:03.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2021-09-17T01:38:03.428 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-09-17T01:38:03.437 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-09-17T01:38:03.438 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-09-17T01:38:03.459 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-09-17T01:38:03.461 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi013.front.sepia.ceph.com,smithi045.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-09-17T01:38:03.893 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-09-17T01:38:04.714 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-09-17T01:38:08.008 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-09-17T01:38:08.033 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:08 +0000 (0:00:00.042) 0:00:00.042 ****** 2021-09-17T01:38:08.046 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:08 +0000 (0:00:00.013) 0:00:00.055 ****** 2021-09-17T01:38:09.624 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi045.front.sepia.ceph.com] 2021-09-17T01:38:09.672 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:09 +0000 (0:00:01.626) 0:00:01.682 ****** 2021-09-17T01:38:09.950 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-17T01:38:09.973 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:09 +0000 (0:00:00.300) 0:00:01.983 ****** 2021-09-17T01:38:10.374 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-09-17T01:38:10.376 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-09-17T01:38:10.396 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:10 +0000 (0:00:00.423) 0:00:02.406 ****** 2021-09-17T01:38:10.465 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-09-17T01:38:10.467 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:10.488 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:10 +0000 (0:00:00.091) 0:00:02.498 ****** 2021-09-17T01:38:10.551 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi013.front.sepia.ceph.com] 2021-09-17T01:38:10.598 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:10 +0000 (0:00:00.110) 0:00:02.608 ****** 2021-09-17T01:38:10.751 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-09-17T01:38:10.753 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T01:38:10.774 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:10 +0000 (0:00:00.175) 0:00:02.784 ****** 2021-09-17T01:38:10.791 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-09-17T01:38:10.793 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-17T01:38:10.816 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:10 +0000 (0:00:00.041) 0:00:02.825 ****** 2021-09-17T01:38:11.119 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** ok: [smithi013.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-09-17T01:38:11.149 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:11 +0000 (0:00:00.332) 0:00:03.158 ****** 2021-09-17T01:38:11.232 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:11.254 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:11 +0000 (0:00:00.105) 0:00:03.264 ****** 2021-09-17T01:38:11.453 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi013.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T01:38:11.475 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:11 +0000 (0:00:00.221) 0:00:03.485 ****** 2021-09-17T01:38:11.691 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1631842690.8320615-31620-214476924331744/source", "state": "file", "uid": 0} 2021-09-17T01:38:11.719 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:11 +0000 (0:00:00.243) 0:00:03.729 ****** 2021-09-17T01:38:12.092 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1631842691.524682-31717-253770691207445/source", "state": "file", "uid": 0} 2021-09-17T01:38:12.116 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:12 +0000 (0:00:00.397) 0:00:04.126 ****** 2021-09-17T01:38:12.136 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-09-17T01:38:12.138 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-09-17T01:38:12.140 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-09-17T01:38:12.445 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-09-17T01:38:12.447 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-09-17T01:38:12.510 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-09-17T01:38:12.511 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-17T01:38:12.524 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Friday 17 September 2021 01:38:12 +0000 (0:00:00.407) 0:00:04.534 ****** 2021-09-17T01:38:13.059 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] 2021-09-17T01:38:13.078 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] 2021-09-17T01:38:13.097 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** 2021-09-17T01:38:13.099 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:13 +0000 (0:00:00.572) 0:00:05.106 ****** 2021-09-17T01:38:13.131 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-17T01:38:13.141 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-17T01:38:13.159 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-17T01:38:13.325 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:13 +0000 (0:00:00.227) 0:00:05.333 ****** 2021-09-17T01:38:13.348 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:13 +0000 (0:00:00.023) 0:00:05.357 ****** 2021-09-17T01:38:13.989 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi013.front.sepia.ceph.com] 2021-09-17T01:38:14.019 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] 2021-09-17T01:38:14.048 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-09-17T01:38:14.050 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:14 +0000 (0:00:00.700) 0:00:06.057 ****** 2021-09-17T01:38:14.094 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:14 +0000 (0:00:00.045) 0:00:06.103 ****** 2021-09-17T01:38:14.709 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-09-17T01:38:14.711 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.187135", "end": "2021-09-17 01:38:14.684388", "rc": 0, "start": "2021-09-17 01:38:14.497253", "stderr": "", "stderr_lines": [], "stdout": "39f530053dd0894c844c464d403644cc64956570", "stdout_lines": ["39f530053dd0894c844c464d403644cc64956570"]} changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.182860", "end": "2021-09-17 01:38:14.690075", "rc": 0, "start": "2021-09-17 01:38:14.507215", "stderr": "", "stderr_lines": [], "stdout": "39f530053dd0894c844c464d403644cc64956570", "stdout_lines": ["39f530053dd0894c844c464d403644cc64956570"]} 2021-09-17T01:38:14.731 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:14 +0000 (0:00:00.637) 0:00:06.741 ****** 2021-09-17T01:38:14.751 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:14 +0000 (0:00:00.019) 0:00:06.760 ****** 2021-09-17T01:38:14.945 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-09-17T01:38:14.947 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013191", "end": "2021-09-17 01:38:14.920796", "failed_when_result": false, "rc": 0, "start": "2021-09-17 01:38:14.907605", "stderr": "", "stderr_lines": [], "stdout": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "stdout_lines": ["b2a91f5f527dcc382f8ed49697328a42af627ee8"]} 2021-09-17T01:38:14.967 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:14 +0000 (0:00:00.215) 0:00:06.976 ****** 2021-09-17T01:38:14.974 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.006269", "end": "2021-09-17 01:38:14.933342", "failed_when_result": false, "rc": 0, "start": "2021-09-17 01:38:14.927073", "stderr": "", "stderr_lines": [], "stdout": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "stdout_lines": ["b2a91f5f527dcc382f8ed49697328a42af627ee8"]} 2021-09-17T01:38:14.999 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:14 +0000 (0:00:00.032) 0:00:07.008 ****** 2021-09-17T01:38:15.036 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-09-17T01:38:15.038 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:15.060 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.061) 0:00:07.070 ****** 2021-09-17T01:38:15.077 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:15.100 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.039) 0:00:07.109 ****** 2021-09-17T01:38:15.127 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-09-17T01:38:15.129 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:15.150 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.050) 0:00:07.160 ****** 2021-09-17T01:38:15.170 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:15.193 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.041) 0:00:07.202 ****** 2021-09-17T01:38:15.220 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-09-17T01:38:15.222 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:15.243 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.050) 0:00:07.252 ****** 2021-09-17T01:38:15.262 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:15.285 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.042) 0:00:07.295 ****** 2021-09-17T01:38:15.312 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-09-17T01:38:15.314 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:15.335 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.049) 0:00:07.345 ****** 2021-09-17T01:38:15.363 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:15.385 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.050) 0:00:07.395 ****** 2021-09-17T01:38:15.412 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-09-17T01:38:15.414 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:15.436 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.049) 0:00:07.445 ****** 2021-09-17T01:38:15.464 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:38:15.486 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.050) 0:00:07.495 ****** 2021-09-17T01:38:15.695 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-09-17T01:38:15.697 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T01:38:15.718 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.231) 0:00:07.727 ****** 2021-09-17T01:38:15.738 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T01:38:15.769 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:38:15 +0000 (0:00:00.050) 0:00:07.778 ****** 2021-09-17T01:38:16.025 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-09-17T01:38:16.027 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-17T01:38:16.072 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-17T01:38:16.232 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-17T01:38:16.276 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-17T01:38:16.433 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-17T01:38:16.476 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-17T01:38:16.643 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-17T01:38:16.686 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-17T01:38:16.844 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-17T01:38:16.887 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-17T01:38:17.054 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-17T01:38:17.088 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-17T01:38:17.256 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-17T01:38:17.288 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-17T01:38:17.455 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-17T01:38:17.491 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-17T01:38:17.660 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-17T01:38:17.702 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-17T01:38:17.869 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-17T01:38:17.911 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-17T01:38:18.068 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-17T01:38:18.110 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-17T01:38:18.269 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-17T01:38:18.322 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-17T01:38:18.480 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-17T01:38:18.522 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-17T01:38:18.689 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-17T01:38:18.732 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-17T01:38:18.900 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-17T01:38:18.933 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-17T01:38:19.101 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-17T01:38:19.143 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-17T01:38:19.311 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-17T01:38:19.354 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-17T01:38:19.520 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-17T01:38:19.552 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-17T01:38:19.730 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-17T01:38:19.762 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-17T01:38:19.939 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-17T01:38:19.961 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-17T01:38:20.138 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-17T01:38:20.171 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-17T01:38:20.349 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-17T01:38:20.371 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-17T01:38:20.548 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-17T01:38:20.580 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-17T01:38:20.757 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-17T01:38:20.779 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-17T01:38:20.957 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-17T01:38:20.990 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-17T01:38:21.158 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-17T01:38:21.191 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-17T01:38:21.369 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-17T01:38:21.392 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-17T01:38:21.571 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-17T01:38:21.594 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-17T01:38:21.775 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-17T01:38:21.798 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-17T01:38:21.975 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-17T01:38:22.007 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-17T01:38:22.187 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-17T01:38:22.209 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-17T01:38:22.388 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-17T01:38:22.420 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-17T01:38:22.597 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-17T01:38:22.619 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-17T01:38:22.798 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-09-17T01:38:22.821 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-09-17T01:38:22.999 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-17T01:38:23.031 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-17T01:38:23.199 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-17T01:38:23.232 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-17T01:38:23.411 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-17T01:38:23.433 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-17T01:38:23.611 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-17T01:38:23.633 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-17T01:38:23.812 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-17T01:38:23.845 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-17T01:38:24.023 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-17T01:38:24.045 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-17T01:38:24.224 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-17T01:38:24.256 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-17T01:38:24.434 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-17T01:38:24.456 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-17T01:38:24.635 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-17T01:38:24.657 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-17T01:38:24.836 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-17T01:38:24.870 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-17T01:38:25.050 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-17T01:38:25.072 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-17T01:38:25.263 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-17T01:38:25.272 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-17T01:38:25.471 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-17T01:38:25.484 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-17T01:38:25.673 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-17T01:38:25.696 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-17T01:38:25.886 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-17T01:38:25.897 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-17T01:38:26.086 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-17T01:38:26.097 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-17T01:38:26.297 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-17T01:38:26.299 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-17T01:38:26.496 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-17T01:38:26.509 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-17T01:38:26.709 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-17T01:38:26.711 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-17T01:38:26.909 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-17T01:38:26.911 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-17T01:38:27.111 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-17T01:38:27.122 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-17T01:38:27.321 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-17T01:38:27.322 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-17T01:38:27.523 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-17T01:38:27.535 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-17T01:38:27.732 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-17T01:38:27.734 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-17T01:38:27.933 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-17T01:38:27.944 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-17T01:38:28.132 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-17T01:38:28.144 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-17T01:38:28.342 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-17T01:38:28.354 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-17T01:38:28.543 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-17T01:38:28.555 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-17T01:38:28.753 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-17T01:38:28.755 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-17T01:38:28.963 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-17T01:38:28.965 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-17T01:38:29.162 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-17T01:38:29.163 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-17T01:38:29.362 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-17T01:38:29.374 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-17T01:38:29.562 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-17T01:38:29.574 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-17T01:38:29.775 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-17T01:38:29.776 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-17T01:38:29.975 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-17T01:38:29.977 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-17T01:38:30.176 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-17T01:38:30.188 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-17T01:38:30.385 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-17T01:38:30.397 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-17T01:38:30.584 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-17T01:38:30.596 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-17T01:38:30.796 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-17T01:38:30.963 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-17T01:38:31.007 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-17T01:38:31.009 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-17T01:38:31.217 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-17T01:38:31.218 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-17T01:38:31.416 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-17T01:38:31.417 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-17T01:38:31.615 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-17T01:38:31.627 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-17T01:38:31.825 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-17T01:38:31.826 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-17T01:38:32.024 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-17T01:38:32.035 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-17T01:38:32.235 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-17T01:38:32.237 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-17T01:38:32.433 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-09-17T01:38:32.445 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-09-17T01:38:32.634 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-17T01:38:32.644 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-17T01:38:32.842 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-17T01:38:32.843 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-17T01:38:33.041 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-17T01:38:33.052 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-17T01:38:33.251 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-17T01:38:33.252 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-17T01:38:33.450 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-17T01:38:33.462 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-17T01:38:33.660 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-17T01:38:33.661 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-17T01:38:33.862 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-17T01:38:33.976 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-17T01:38:34.061 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-17T01:38:34.073 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-17T01:38:34.261 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-17T01:38:34.273 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-17T01:38:34.471 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-17T01:38:34.482 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-17T01:38:34.671 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-17T01:38:34.682 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-17T01:38:34.881 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-17T01:38:34.882 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-17T01:38:35.079 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-17T01:38:35.185 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-17T01:38:35.288 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-17T01:38:35.299 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-17T01:38:35.488 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-17T01:38:35.500 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-17T01:38:35.700 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-17T01:38:35.702 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-17T01:38:35.901 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-17T01:38:35.912 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-17T01:38:36.113 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-17T01:38:36.114 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-17T01:38:36.317 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-17T01:38:36.319 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-17T01:38:36.552 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-17T01:38:36.564 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-17T01:38:36.779 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-17T01:38:36.784 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-17T01:38:36.994 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-17T01:38:37.017 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-17T01:38:37.199 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-17T01:38:37.274 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-17T01:38:37.423 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-17T01:38:37.489 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-17T01:38:37.651 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-17T01:38:37.705 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-17T01:38:37.876 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-17T01:38:37.918 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-17T01:38:38.109 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-17T01:38:38.145 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-17T01:38:38.147 INFO:teuthology.task.ansible.out: 2021-09-17T01:38:38.319 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-17T01:38:38.352 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-17T01:38:38.533 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-17T01:38:38.565 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-17T01:38:38.733 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-17T01:38:38.776 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-17T01:38:38.945 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-17T01:38:38.990 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-17T01:38:39.145 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-17T01:38:39.187 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-17T01:38:39.355 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-17T01:38:39.397 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-17T01:38:39.557 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-17T01:38:39.600 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-17T01:38:39.769 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-17T01:38:39.816 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-17T01:38:39.973 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-17T01:38:40.017 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-17T01:38:40.188 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-17T01:38:40.221 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-17T01:38:40.390 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-17T01:38:40.434 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-17T01:38:40.604 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-17T01:38:40.635 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-17T01:38:40.804 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-17T01:38:40.835 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-17T01:38:41.024 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-17T01:38:41.046 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-17T01:38:41.236 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-17T01:38:41.247 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-17T01:38:41.447 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-17T01:38:41.459 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-17T01:38:41.650 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-17T01:38:41.662 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-17T01:38:41.858 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-17T01:38:41.869 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-17T01:38:42.058 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-09-17T01:38:42.071 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-09-17T01:38:42.260 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-09-17T01:38:42.271 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-09-17T01:38:42.463 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-17T01:38:42.485 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-17T01:38:42.674 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-17T01:38:42.686 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-17T01:38:42.875 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-17T01:38:42.888 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-17T01:38:43.077 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-17T01:38:43.089 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-17T01:38:43.276 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-17T01:38:43.298 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-17T01:38:44.901 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-09-17T01:38:44.976 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-09-17T01:38:46.684 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-09-17T01:38:46.769 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-09-17T01:38:48.248 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-09-17T01:38:48.446 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-09-17T01:38:49.834 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-09-17T01:38:50.085 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-09-17T01:38:51.442 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-09-17T01:38:51.683 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-09-17T01:38:53.087 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-09-17T01:38:53.295 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-09-17T01:38:54.820 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-09-17T01:38:54.925 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-09-17T01:38:56.504 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-09-17T01:38:56.567 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-09-17T01:38:58.143 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-09-17T01:38:58.206 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-09-17T01:38:59.765 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-09-17T01:38:59.901 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-09-17T01:39:01.402 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-09-17T01:39:01.580 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-09-17T01:39:03.056 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-09-17T01:39:03.275 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-09-17T01:39:04.710 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-09-17T01:39:04.920 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-09-17T01:39:06.387 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-09-17T01:39:06.616 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-09-17T01:39:08.075 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-09-17T01:39:08.283 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-09-17T01:39:09.884 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-09-17T01:39:10.011 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-09-17T01:39:11.636 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-09-17T01:39:11.700 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-09-17T01:39:13.429 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-09-17T01:39:13.441 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-09-17T01:39:15.222 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-09-17T01:39:15.320 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-09-17T01:39:17.054 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-09-17T01:39:17.080 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-09-17T01:39:17.273 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:17 +0000 (0:01:01.503) 0:01:09.282 ****** 2021-09-17T01:39:17.293 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:17 +0000 (0:00:00.020) 0:01:09.303 ****** 2021-09-17T01:39:17.338 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-09-17T01:39:17.362 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:17 +0000 (0:00:00.068) 0:01:09.371 ****** 2021-09-17T01:39:17.395 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:17 +0000 (0:00:00.033) 0:01:09.405 ****** 2021-09-17T01:39:17.620 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-09-17T01:39:17.622 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-09-17T01:39:17.624 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-09-17T01:39:17.625 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed208 2021-09-17T01:39:17.627 INFO:teuthology.task.ansible.out:2caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": 2021-09-17T01:39:17.628 INFO:teuthology.task.ansible.out: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vas 2021-09-17T01:39:17.629 INFO:teuthology.task.ansible.out:u", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d 2021-09-17T01:39:17.630 INFO:teuthology.task.ansible.out:3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDe 2021-09-17T01:39:17.632 INFO:teuthology.task.ansible.out:vAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBM 2021-09-17T01:39:17.634 INFO:teuthology.task.ansible.out:gw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmeri 2021-09-17T01:39:17.635 INFO:teuthology.task.ansible.out:cp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361 2021-09-17T01:39:17.637 INFO:teuthology.task.ansible.out:f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9 2021-09-17T01:39:17.638 INFO:teuthology.task.ansible.out:JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": " 2021-09-17T01:39:17.640 INFO:teuthology.task.ansible.out:shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f7 2021-09-17T01:39:17.642 INFO:teuthology.task.ansible.out:0cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore 2021-09-17T01:39:17.643 INFO:teuthology.task.ansible.out: gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142 2021-09-17T01:39:17.645 INFO:teuthology.task.ansible.out:120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}]}, "changed": false} 2021-09-17T01:39:17.669 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:17 +0000 (0:00:00.273) 0:01:09.678 ****** 2021-09-17T01:39:17.681 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-09-17T01:39:17.683 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-09-17T01:39:17.684 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed208 2021-09-17T01:39:17.686 INFO:teuthology.task.ansible.out:2caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": 2021-09-17T01:39:17.688 INFO:teuthology.task.ansible.out: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d 2021-09-17T01:39:17.689 INFO:teuthology.task.ansible.out:3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDe 2021-09-17T01:39:17.691 INFO:teuthology.task.ansible.out:vAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBM 2021-09-17T01:39:17.693 INFO:teuthology.task.ansible.out:gw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmeri 2021-09-17T01:39:17.694 INFO:teuthology.task.ansible.out:cp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361 2021-09-17T01:39:17.696 INFO:teuthology.task.ansible.out:f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9 2021-09-17T01:39:17.697 INFO:teuthology.task.ansible.out:JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa15674 2021-09-17T01:39:17.698 INFO:teuthology.task.ansible.out:32aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": " 2021-09-17T01:39:17.699 INFO:teuthology.task.ansible.out:shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6 2021-09-17T01:39:17.701 INFO:teuthology.task.ansible.out:878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovp 2021-09-17T01:39:17.702 INFO:teuthology.task.ansible.out:n": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}]}, "changed": false} 2021-09-17T01:39:17.725 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:17 +0000 (0:00:00.056) 0:01:09.735 ****** 2021-09-17T01:39:18.638 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-09-17T01:39:18.640 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"after": "39f530053dd0894c844c464d403644cc64956570", "attempts": 1, "before": "39f530053dd0894c844c464d403644cc64956570", "changed": false, "remote_url_changed": false} ok: [smithi013.front.sepia.ceph.com] => {"after": "39f530053dd0894c844c464d403644cc64956570", "attempts": 1, "before": "39f530053dd0894c844c464d403644cc64956570", "changed": false, "remote_url_changed": false} 2021-09-17T01:39:18.660 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:18 +0000 (0:00:00.934) 0:01:10.670 ****** 2021-09-17T01:39:18.679 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:18 +0000 (0:00:00.018) 0:01:10.689 ****** 2021-09-17T01:39:19.161 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-09-17T01:39:19.185 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-09-17T01:39:19.399 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-17T01:39:19.432 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-17T01:39:19.690 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-17T01:39:19.691 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-17T01:39:19.902 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-17T01:39:19.903 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-17T01:39:19.925 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-17T01:39:19.927 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-17T01:39:20.160 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-17T01:39:20.171 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-17T01:39:20.401 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-17T01:39:20.402 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-17T01:39:20.412 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-17T01:39:20.414 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-17T01:39:20.632 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-17T01:39:20.654 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-17T01:39:20.894 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-17T01:39:20.916 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-17T01:39:21.139 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-09-17T01:39:21.150 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:21.161 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-09-17T01:39:21.173 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:21.415 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-17T01:39:21.417 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-17T01:39:21.426 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-17T01:39:21.428 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-17T01:39:21.657 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-17T01:39:21.659 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-09-17T01:39:21.668 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:21.670 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-17T01:39:21.671 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:21.680 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:21.691 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:21.913 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-17T01:39:21.945 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-17T01:39:22.164 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-17T01:39:22.166 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-17T01:39:22.197 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-17T01:39:22.199 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-17T01:39:22.397 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-17T01:39:22.441 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-17T01:39:22.653 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-17T01:39:22.697 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-17T01:39:22.896 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-17T01:39:22.939 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-17T01:39:23.137 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-17T01:39:23.139 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-17T01:39:23.182 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-17T01:39:23.184 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-17T01:39:23.383 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-09-17T01:39:23.385 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-09-17T01:39:23.436 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-09-17T01:39:23.438 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-09-17T01:39:23.627 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-17T01:39:23.691 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-17T01:39:23.871 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-17T01:39:23.935 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-17T01:39:24.114 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-09-17T01:39:24.116 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-17T01:39:24.178 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-09-17T01:39:24.180 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-17T01:39:24.366 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-09-17T01:39:24.418 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-09-17T01:39:24.608 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-17T01:39:24.661 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-17T01:39:24.840 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-17T01:39:24.904 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-17T01:39:25.083 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-17T01:39:25.158 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-17T01:39:25.346 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-17T01:39:25.399 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-17T01:39:25.586 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-17T01:39:25.588 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-17T01:39:25.650 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-17T01:39:25.652 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-17T01:39:25.832 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-17T01:39:25.896 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-17T01:39:26.075 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-17T01:39:26.139 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-17T01:39:26.327 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-17T01:39:26.380 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-17T01:39:26.570 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-17T01:39:26.622 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-17T01:39:26.811 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-17T01:39:26.864 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-17T01:39:27.043 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-17T01:39:27.107 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-17T01:39:27.295 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-17T01:39:27.348 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-17T01:39:27.540 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-09-17T01:39:27.541 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-09-17T01:39:27.594 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-09-17T01:39:27.596 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-09-17T01:39:27.783 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-17T01:39:27.784 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-17T01:39:27.836 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-17T01:39:27.838 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-17T01:39:28.026 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-17T01:39:28.080 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-17T01:39:28.271 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-17T01:39:28.343 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-17T01:39:28.502 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-17T01:39:28.577 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-17T01:39:28.747 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-17T01:39:28.749 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-17T01:39:28.822 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-17T01:39:28.825 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-17T01:39:29.013 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-09-17T01:39:29.014 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-09-17T01:39:29.016 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-09-17T01:39:29.076 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-09-17T01:39:29.078 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-09-17T01:39:29.080 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-09-17T01:39:29.255 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-17T01:39:29.311 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-17T01:39:29.501 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-17T01:39:29.554 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-17T01:39:29.743 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-17T01:39:29.795 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-17T01:39:29.984 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-17T01:39:30.038 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-17T01:39:30.227 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-17T01:39:30.291 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-17T01:39:30.470 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-09-17T01:39:30.476 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-09-17T01:39:30.534 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-09-17T01:39:30.535 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-09-17T01:39:30.716 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-17T01:39:30.789 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-17T01:39:30.956 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-17T01:39:31.031 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-17T01:39:31.205 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-17T01:39:31.207 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-17T01:39:31.268 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-17T01:39:31.269 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-17T01:39:31.447 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-17T01:39:31.510 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-17T01:39:31.690 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-17T01:39:31.755 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-17T01:39:31.943 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-17T01:39:31.945 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-17T01:39:31.997 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-17T01:39:31.999 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-17T01:39:32.174 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-17T01:39:32.242 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-17T01:39:32.421 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-17T01:39:32.487 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-17T01:39:32.675 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-17T01:39:32.728 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-17T01:39:32.909 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-17T01:39:32.974 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-17T01:39:33.162 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-17T01:39:33.226 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-17T01:39:33.403 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-17T01:39:33.466 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-17T01:39:33.644 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-17T01:39:33.708 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-17T01:39:33.888 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-17T01:39:33.951 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-17T01:39:34.132 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-17T01:39:34.186 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-17T01:39:34.375 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-17T01:39:34.429 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-17T01:39:34.618 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-17T01:39:34.682 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-17T01:39:34.873 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-17T01:39:34.926 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-17T01:39:35.105 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-17T01:39:35.168 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-17T01:39:35.347 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-17T01:39:35.410 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-17T01:39:35.588 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-17T01:39:35.652 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-17T01:39:35.829 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-17T01:39:35.892 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-17T01:39:36.070 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-17T01:39:36.150 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-17T01:39:36.307 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-17T01:39:36.391 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-17T01:39:36.549 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-17T01:39:36.634 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-17T01:39:36.791 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-17T01:39:36.875 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-17T01:39:37.033 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-17T01:39:37.035 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-17T01:39:37.116 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-17T01:39:37.119 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-17T01:39:37.273 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-17T01:39:37.358 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-17T01:39:37.514 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-17T01:39:37.600 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-17T01:39:37.761 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-17T01:39:37.857 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-17T01:39:38.006 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-17T01:39:38.008 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-17T01:39:38.101 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-17T01:39:38.102 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-17T01:39:38.247 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-17T01:39:38.348 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-17T01:39:38.532 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-17T01:39:38.603 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-17T01:39:38.796 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-17T01:39:38.864 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-17T01:39:39.093 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-17T01:39:39.147 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-17T01:39:39.379 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-17T01:39:39.412 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-17T01:39:39.648 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-17T01:39:39.670 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-17T01:39:39.902 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-17T01:39:39.913 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-17T01:39:40.150 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-17T01:39:40.152 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-17T01:39:40.162 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-17T01:39:40.163 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-17T01:39:40.393 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-17T01:39:40.403 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-17T01:39:40.636 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-17T01:39:40.649 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-17T01:39:40.888 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-17T01:39:40.891 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-17T01:39:41.136 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": 2021-09-17T01:39:41.138 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-09-17T01:39:41.366 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-17T01:39:41.388 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-17T01:39:41.617 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-17T01:39:41.640 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-17T01:39:41.860 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-17T01:39:41.862 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-17T01:39:41.882 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-17T01:39:41.884 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-17T01:39:42.091 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-17T01:39:42.134 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-17T01:39:42.337 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-17T01:39:42.380 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-17T01:39:42.591 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-17T01:39:42.612 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-17T01:39:42.834 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-17T01:39:42.854 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-17T01:39:43.076 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-17T01:39:43.096 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-17T01:39:43.317 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-17T01:39:43.339 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-17T01:39:43.550 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-17T01:39:43.582 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-17T01:39:43.793 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-17T01:39:43.836 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-17T01:39:44.038 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-17T01:39:44.080 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-17T01:39:44.283 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-17T01:39:44.330 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-17T01:39:44.532 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-17T01:39:44.584 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-17T01:39:44.774 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-17T01:39:44.827 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-17T01:39:45.016 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-17T01:39:45.079 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-17T01:39:45.257 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-17T01:39:45.320 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-17T01:39:45.498 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-09-17T01:39:45.551 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-09-17T01:39:45.739 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-17T01:39:45.802 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-17T01:39:45.981 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-17T01:39:46.046 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-17T01:39:46.228 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-17T01:39:46.282 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-17T01:39:46.470 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-17T01:39:46.472 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-17T01:39:46.536 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-17T01:39:46.537 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-17T01:39:46.715 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-17T01:39:46.779 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-17T01:39:46.957 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-17T01:39:47.012 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-17T01:39:47.205 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-17T01:39:47.258 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-17T01:39:47.438 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-17T01:39:47.501 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-17T01:39:47.682 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-17T01:39:47.760 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-17T01:39:47.942 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-17T01:39:48.008 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-17T01:39:48.176 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-17T01:39:48.262 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-17T01:39:48.422 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-17T01:39:48.496 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-17T01:39:48.663 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-17T01:39:48.737 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-17T01:39:48.906 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-17T01:39:48.980 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-17T01:39:49.158 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-17T01:39:49.231 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-17T01:39:49.398 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-17T01:39:49.471 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-17T01:39:49.651 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-17T01:39:49.726 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-17T01:39:49.894 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-17T01:39:49.978 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-17T01:39:50.147 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-17T01:39:50.233 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-17T01:39:50.391 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-17T01:39:50.393 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-17T01:39:50.476 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-17T01:39:50.477 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-17T01:39:50.636 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-17T01:39:50.722 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-17T01:39:50.880 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-17T01:39:50.965 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-17T01:39:51.131 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-17T01:39:51.216 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-17T01:39:51.385 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-17T01:39:51.459 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-17T01:39:51.628 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-17T01:39:51.703 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-17T01:39:51.871 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-17T01:39:51.966 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-17T01:39:52.114 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-17T01:39:52.219 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-17T01:39:52.355 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-17T01:39:52.463 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-17T01:39:52.610 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-17T01:39:52.706 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-17T01:39:52.853 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-17T01:39:52.948 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-17T01:39:53.094 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-17T01:39:53.188 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-17T01:39:53.344 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-17T01:39:53.438 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-17T01:39:53.584 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-17T01:39:53.690 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-17T01:39:53.836 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-17T01:39:53.919 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-17T01:39:54.076 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-17T01:39:54.169 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-17T01:39:54.325 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-17T01:39:54.412 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-17T01:39:54.568 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-17T01:39:54.672 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-17T01:39:54.818 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-17T01:39:54.913 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-17T01:39:55.062 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-17T01:39:55.158 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-17T01:39:55.306 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-17T01:39:55.402 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-17T01:39:55.548 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-17T01:39:55.644 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-17T01:39:55.800 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-09-17T01:39:55.887 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:55 +0000 (0:00:37.207) 0:01:47.896 ****** 2021-09-17T01:39:55.906 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-09-17T01:39:55.993 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:55 +0000 (0:00:00.106) 0:01:48.003 ****** 2021-09-17T01:39:56.002 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-09-17T01:39:56.003 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.004 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.005 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.020 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.021 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.022 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.033 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.076 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.087 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.089 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.100 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.111 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.113 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.129 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.140 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.278 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-17T01:39:56.290 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.292 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.381 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-17T01:39:56.392 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.414 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.530 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-17T01:39:56.649 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-17T01:39:56.779 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfad 2021-09-17T01:39:56.781 INFO:teuthology.task.ansible.out:a28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.790 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.792 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.802 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.813 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.815 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.825 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.836 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.838 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.848 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.849 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.859 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.871 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.886 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.887 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.897 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.898 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-09-17T01:39:56.899 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916 2021-09-17T01:39:56.900 INFO:teuthology.task.ansible.out:716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.913 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.915 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.925 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.937 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.939 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.949 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.951 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.952 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'a 2021-09-17T01:39:56.953 INFO:teuthology.task.ansible.out:bhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.964 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.967 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.977 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.979 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.980 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.981 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.982 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.993 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:56.995 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.006 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.007 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.018 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.020 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.030 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.033 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.045 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.057 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d5 2021-09-17T01:39:57.059 INFO:teuthology.task.ansible.out:7e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.068 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.069 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.080 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.081 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.093 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.094 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.104 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.106 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.116 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.117 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.129 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.130 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.141 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.143 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.154 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.157 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.166 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.168 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.179 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.180 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.191 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.203 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.204 INFO:teuthology.task.ansible.out: skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.206 INFO:teuthology.task.ansible.out: skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.206 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.207 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@ 2021-09-17T01:39:57.208 INFO:teuthology.task.ansible.out:localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.221 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.223 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.233 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.235 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.235 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.237 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.247 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.249 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.250 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.265 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.278 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.280 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.290 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.292 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.302 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.315 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.329 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.331 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.341 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.343 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.343 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.354 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.356 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.367 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.368 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.369 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.380 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.381 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.392 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.393 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.394 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.394 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.405 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.417 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.427 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.429 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.442 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.443 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.443 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.454 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.456 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.457 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", " 2021-09-17T01:39:57.458 INFO:teuthology.task.ansible.out:ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.467 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.469 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.479 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.482 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.483 INFO:teuthology.task.ansible.out: skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.495 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.497 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.507 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.509 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.509 INFO:teuthology.task.ansible.out: 2021-09-17T01:39:57.524 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.526 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.526 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.528 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.529 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.540 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.541 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.542 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.554 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.555 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.566 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.568 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.578 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.579 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.591 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.602 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.604 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.617 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.629 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": 2021-09-17T01:39:57.631 INFO:teuthology.task.ansible.out:{"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.642 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.653 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.654 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.669 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.683 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": " 2021-09-17T01:39:57.685 INFO:teuthology.task.ansible.out:item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.692 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.693 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.695 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.697 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.719 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.727 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.747 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.770 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:57 +0000 (0:00:01.776) 0:01:49.779 ****** 2021-09-17T01:39:57.777 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.779 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.780 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.793 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.795 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.805 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.807 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.817 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.830 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.832 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.844 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.845 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.868 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:57 +0000 (0:00:00.098) 0:01:49.877 ****** 2021-09-17T01:39:57.877 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.880 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.882 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:57.927 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:57 +0000 (0:00:00.058) 0:01:49.936 ****** 2021-09-17T01:39:57.998 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-09-17T01:39:58.000 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:39:58.020 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:39:58 +0000 (0:00:00.093) 0:01:50.030 ****** 2021-09-17T01:39:58.154 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-09-17T01:39:58.155 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-17T01:39:58.311 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-17T01:39:58.353 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-17T01:39:58.510 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-17T01:39:58.564 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-17T01:39:58.713 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-17T01:39:58.754 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-17T01:39:58.922 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-17T01:39:58.955 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-17T01:39:59.124 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-17T01:39:59.157 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-17T01:39:59.317 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-17T01:39:59.349 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-17T01:39:59.516 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-17T01:39:59.559 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-17T01:39:59.726 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-17T01:39:59.758 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-17T01:39:59.928 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-17T01:39:59.949 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-17T01:40:00.130 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-17T01:40:00.164 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-17T01:40:00.332 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-17T01:40:00.366 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-17T01:40:00.527 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-17T01:40:00.560 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-17T01:40:00.728 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-17T01:40:00.760 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-17T01:40:00.930 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-17T01:40:00.961 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-17T01:40:01.135 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-17T01:40:01.167 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-17T01:40:01.335 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-17T01:40:01.369 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-17T01:40:01.540 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-17T01:40:01.576 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-17T01:40:01.747 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-17T01:40:01.776 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-17T01:40:01.948 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-17T01:40:01.981 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-17T01:40:02.150 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-17T01:40:02.194 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-17T01:40:02.368 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-17T01:40:02.402 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-17T01:40:02.578 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-17T01:40:02.611 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-17T01:40:02.791 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-17T01:40:02.813 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-17T01:40:03.006 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-17T01:40:03.017 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-17T01:40:03.220 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-17T01:40:03.222 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-17T01:40:03.423 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-17T01:40:03.426 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-17T01:40:03.626 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-17T01:40:03.638 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-17T01:40:03.831 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-17T01:40:03.845 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-17T01:40:04.036 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-17T01:40:04.048 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-17T01:40:04.252 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-17T01:40:04.255 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-17T01:40:04.456 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-17T01:40:04.457 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-17T01:40:04.668 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-17T01:40:04.686 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-17T01:40:04.878 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-17T01:40:04.892 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-17T01:40:05.086 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-17T01:40:05.108 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-17T01:40:05.288 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-17T01:40:05.320 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-17T01:40:05.491 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-17T01:40:05.525 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-17T01:40:05.697 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-17T01:40:05.729 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-17T01:40:05.902 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-17T01:40:05.932 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-17T01:40:06.141 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-17T01:40:07.565 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-09-17T01:40:08.070 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-09-17T01:40:09.236 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-09-17T01:40:09.784 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-09-17T01:40:10.929 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-09-17T01:40:11.484 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-09-17T01:40:12.633 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-09-17T01:40:13.215 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-09-17T01:40:14.284 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-09-17T01:40:14.891 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-09-17T01:40:15.917 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-09-17T01:40:16.129 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-17T01:40:16.341 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-17T01:40:16.542 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-17T01:40:16.554 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-09-17T01:40:16.744 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-17T01:40:16.787 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-17T01:40:16.946 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-17T01:40:16.998 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-17T01:40:17.136 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-17T01:40:17.190 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-17T01:40:17.350 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-17T01:40:17.393 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-17T01:40:17.540 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-17T01:40:17.593 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-17T01:40:17.738 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-17T01:40:17.791 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-17T01:40:17.937 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-17T01:40:17.994 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:17 +0000 (0:00:19.973) 0:02:10.003 ****** 2021-09-17T01:40:18.003 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-17T01:40:18.204 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-17T01:40:18.403 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-17T01:40:18.552 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-09-17T01:40:18.554 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "checksum": "4a5787f42b17837e1416d13c4325647102121755", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "c85c4869d3df1b2c6c21c5a0709ecc0c", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1631842818.086078-14076-6975500452347/source", "state": "file", "uid": 0} 2021-09-17T01:40:18.580 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:18 +0000 (0:00:00.585) 0:02:10.589 ****** 2021-09-17T01:40:18.609 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-09-17T01:40:18.613 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-17T01:40:18.660 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:18 +0000 (0:00:00.079) 0:02:10.669 ****** 2021-09-17T01:40:18.667 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-09-17T01:40:18.669 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => { "msg": "Host smithi045.front.sepia.ceph.com is running RedHat 8.4 (Ootpa)" } 2021-09-17T01:40:18.694 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:18 +0000 (0:00:00.034) 0:02:10.703 ****** 2021-09-17T01:40:18.777 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-09-17T01:40:18.803 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:18 +0000 (0:00:00.108) 0:02:10.812 ****** 2021-09-17T01:40:18.872 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-09-17T01:40:18.874 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:18.897 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:18 +0000 (0:00:00.093) 0:02:10.906 ****** 2021-09-17T01:40:19.206 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.075672", "end": "2021-09-17 01:40:19.185556", "rc": 0, "start": "2021-09-17 01:40:19.109884", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-09-17T01:40:19.229 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.333) 0:02:11.239 ****** 2021-09-17T01:40:19.281 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-09-17T01:40:19.282 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "checksum": "4a5787f42b17837e1416d13c4325647102121755", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "c85c4869d3df1b2c6c21c5a0709ecc0c", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1631842818.7906678-14159-273351682865101/source", "state": "file", "uid": 0} 2021-09-17T01:40:19.305 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.075) 0:02:11.314 ****** 2021-09-17T01:40:19.313 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-09-17T01:40:19.315 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:19.339 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.033) 0:02:11.348 ****** 2021-09-17T01:40:19.377 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-09-17T01:40:19.379 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => { "msg": "Host smithi013.front.sepia.ceph.com is running RedHat 8.4 (Ootpa)" } 2021-09-17T01:40:19.401 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.062) 0:02:11.410 ****** 2021-09-17T01:40:19.411 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:19.435 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.033) 0:02:11.444 ****** 2021-09-17T01:40:19.490 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-09-17T01:40:19.492 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-09-17T01:40:19.514 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.079) 0:02:11.523 ****** 2021-09-17T01:40:19.521 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-09-17T01:40:19.523 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:19.546 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.032) 0:02:11.555 ****** 2021-09-17T01:40:19.584 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-09-17T01:40:19.586 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:19.610 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.063) 0:02:11.619 ****** 2021-09-17T01:40:19.627 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-09-17T01:40:19.629 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:19.654 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.044) 0:02:11.663 ****** 2021-09-17T01:40:19.737 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:19.760 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.106) 0:02:11.769 ****** 2021-09-17T01:40:19.843 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-09-17T01:40:19.871 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.104) 0:02:11.874 ****** 2021-09-17T01:40:19.935 INFO:teuthology.task.ansible.out: 2021-09-17T01:40:19.937 INFO:teuthology.task.ansible.out:TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.070362", "end": "2021-09-17 01:40:19.900631", "rc": 0, "start": "2021-09-17 01:40:19.830269", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-09-17T01:40:19.965 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:19 +0000 (0:00:00.099) 0:02:11.974 ****** 2021-09-17T01:40:19.985 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-09-17T01:40:19.987 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T01:40:20.009 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:20 +0000 (0:00:00.044) 0:02:12.018 ****** 2021-09-17T01:40:20.038 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-09-17T01:40:20.039 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:20.063 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:20 +0000 (0:00:00.054) 0:02:12.072 ****** 2021-09-17T01:40:20.092 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-09-17T01:40:20.120 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:20 +0000 (0:00:00.056) 0:02:12.129 ****** 2021-09-17T01:40:20.149 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-09-17T01:40:20.151 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:20.177 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:20 +0000 (0:00:00.057) 0:02:12.187 ****** 2021-09-17T01:40:20.271 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-09-17T01:40:20.273 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:20.296 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:20 +0000 (0:00:00.118) 0:02:12.305 ****** 2021-09-17T01:40:20.365 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-09-17T01:40:20.367 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:20.393 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:20 +0000 (0:00:00.096) 0:02:12.402 ****** 2021-09-17T01:40:20.481 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-09-17T01:40:20.483 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:20.513 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:20 +0000 (0:00:00.120) 0:02:12.522 ****** 2021-09-17T01:40:20.606 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-09-17T01:40:20.608 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-09-17T01:40:20.630 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:20 +0000 (0:00:00.117) 0:02:12.639 ****** 2021-09-17T01:40:20.720 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** ok: [smithi013.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T01:40:20.722 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T01:40:20.741 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:20 +0000 (0:00:00.111) 0:02:12.751 ****** 2021-09-17T01:40:20.820 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-09-17T01:40:20.821 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-09-17T01:40:20.841 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:20 +0000 (0:00:00.099) 0:02:12.851 ****** 2021-09-17T01:40:21.421 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1631842821.2669547-jvysa7n7/katello-ca-consumer-latest.noarchlznebt2k.rpm"]} 2021-09-17T01:40:21.444 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:21 +0000 (0:00:00.602) 0:02:13.453 ****** 2021-09-17T01:40:21.884 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1631842821.7326663-whxt14i1/katello-ca-consumer-latest.noarchobgk314f.rpm"]} 2021-09-17T01:40:21.907 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:21 +0000 (0:00:00.463) 0:02:13.916 ****** 2021-09-17T01:40:22.202 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-09-17T01:40:22.204 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "532cb5b70a3f71ce180c165205647e8909d4e8a0", "dest": "/etc/yum/vars/releasever", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/yum/vars/releasever", "secontext": "system_u:object_r:etc_t:s0", "size": 3, "state": "file", "uid": 0} 2021-09-17T01:40:22.230 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:22 +0000 (0:00:00.321) 0:02:14.238 ****** 2021-09-17T01:40:22.441 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "532cb5b70a3f71ce180c165205647e8909d4e8a0", "dest": "/etc/yum/vars/releasever", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/yum/vars/releasever", "secontext": "system_u:object_r:etc_t:s0", "size": 3, "state": "file", "uid": 0} 2021-09-17T01:40:22.465 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:22 +0000 (0:00:00.235) 0:02:14.474 ****** 2021-09-17T01:40:22.905 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** fatal: [smithi045.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-09-17T01:40:22.931 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:22 +0000 (0:00:00.465) 0:02:14.940 ****** 2021-09-17T01:40:23.012 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-09-17T01:40:23.014 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-09-17T01:40:23.035 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:23 +0000 (0:00:00.104) 0:02:15.044 ****** 2021-09-17T01:40:23.146 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** fatal: [smithi013.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-09-17T01:40:23.168 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:23 +0000 (0:00:00.133) 0:02:15.178 ****** 2021-09-17T01:40:23.263 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-09-17T01:40:23.265 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-09-17T01:40:23.287 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:23 +0000 (0:00:00.117) 0:02:15.295 ****** 2021-09-17T01:40:41.491 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-09-17T01:40:41.493 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-17T01:40:41.521 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:41 +0000 (0:00:18.234) 0:02:33.530 ****** 2021-09-17T01:40:41.585 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-17T01:40:41.610 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:41 +0000 (0:00:00.089) 0:02:33.619 ****** 2021-09-17T01:40:41.618 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-09-17T01:40:41.644 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:41 +0000 (0:00:00.034) 0:02:33.653 ****** 2021-09-17T01:40:41.686 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-09-17T01:40:41.710 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:41 +0000 (0:00:00.065) 0:02:33.719 ****** 2021-09-17T01:40:42.342 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-09-17T01:40:42.344 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.454018", "end": "2021-09-17 01:40:42.319848", "failed_when_result": false, "rc": 0, "start": "2021-09-17 01:40:41.865830", "stderr": "", "stderr_lines": [], "stdout": "7.8\n7.9\n7Server\n8\n8.2\n8.3\n8.4", "stdout_lines": ["7.8", "7.9", "7Server", "8", "8.2", "8.3", "8.4"]} 2021-09-17T01:40:42.366 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:42 +0000 (0:00:00.656) 0:02:34.376 ****** 2021-09-17T01:40:42.406 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.448430", "end": "2021-09-17 01:40:42.377960", "failed_when_result": false, "rc": 0, "start": "2021-09-17 01:40:41.929530", "stderr": "", "stderr_lines": [], "stdout": "7.8\n7.9\n7Server\n8\n8.2\n8.3\n8.4", "stdout_lines": ["7.8", "7.9", "7Server", "8", "8.2", "8.3", "8.4"]} 2021-09-17T01:40:42.432 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:42 +0000 (0:00:00.064) 0:02:34.441 ****** 2021-09-17T01:40:42.450 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:42.474 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:42 +0000 (0:00:00.042) 0:02:34.483 ****** 2021-09-17T01:40:42.502 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:42.526 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:42 +0000 (0:00:00.051) 0:02:34.535 ****** 2021-09-17T01:40:42.544 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-09-17T01:40:42.546 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:42.569 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:42 +0000 (0:00:00.042) 0:02:34.578 ****** 2021-09-17T01:40:42.597 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:42.623 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:42 +0000 (0:00:00.054) 0:02:34.633 ****** 2021-09-17T01:40:42.643 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-09-17T01:40:42.643 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T01:40:42.668 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:42 +0000 (0:00:00.044) 0:02:34.677 ****** 2021-09-17T01:40:42.686 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T01:40:42.711 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:42 +0000 (0:00:00.042) 0:02:34.720 ****** 2021-09-17T01:40:42.741 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-17T01:40:42.743 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:42.770 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:42 +0000 (0:00:00.058) 0:02:34.779 ****** 2021-09-17T01:40:42.787 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:42.812 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:42 +0000 (0:00:00.042) 0:02:34.821 ****** 2021-09-17T01:40:45.837 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-09-17T01:40:45.839 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.783170", "end": "2021-09-17 01:40:45.819725", "rc": 0, "start": "2021-09-17 01:40:43.036555", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-09-17T01:40:45.861 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:45 +0000 (0:00:03.048) 0:02:37.870 ****** 2021-09-17T01:40:45.940 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-09-17T01:40:45.962 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:45 +0000 (0:00:00.101) 0:02:37.972 ****** 2021-09-17T01:40:45.969 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-09-17T01:40:45.971 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.936583", "end": "2021-09-17 01:40:45.926612", "rc": 0, "start": "2021-09-17 01:40:42.990029", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-09-17T01:40:45.994 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:45 +0000 (0:00:00.031) 0:02:38.004 ****** 2021-09-17T01:40:46.032 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-09-17T01:40:46.034 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:46.058 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:46 +0000 (0:00:00.061) 0:02:38.066 ****** 2021-09-17T01:40:46.074 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-09-17T01:40:46.076 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-09-17T01:40:46.098 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:46 +0000 (0:00:00.041) 0:02:38.107 ****** 2021-09-17T01:40:46.136 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-09-17T01:40:46.138 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-09-17T01:40:46.166 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:46 +0000 (0:00:00.068) 0:02:38.175 ****** 2021-09-17T01:40:46.177 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-09-17T01:40:46.183 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:46.202 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:46 +0000 (0:00:00.035) 0:02:38.211 ****** 2021-09-17T01:40:46.252 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-09-17T01:40:46.254 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-09-17T01:40:46.284 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:46 +0000 (0:00:00.081) 0:02:38.293 ****** 2021-09-17T01:40:46.291 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-09-17T01:40:46.293 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-09-17T01:40:46.316 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:46 +0000 (0:00:00.032) 0:02:38.325 ****** 2021-09-17T01:40:46.399 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-09-17T01:40:46.421 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:46 +0000 (0:00:00.104) 0:02:38.430 ****** 2021-09-17T01:40:48.740 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** changed: [smithi013.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-17T01:40:48.762 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:48 +0000 (0:00:02.341) 0:02:40.771 ****** 2021-09-17T01:40:48.823 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-17T01:40:48.846 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:48 +0000 (0:00:00.083) 0:02:40.855 ****** 2021-09-17T01:40:52.640 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:03.556542", "end": "2021-09-17 01:40:52.617438", "rc": 0, "start": "2021-09-17 01:40:49.060896", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-09-17T01:40:52.661 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:52 +0000 (0:00:03.815) 0:02:44.671 ****** 2021-09-17T01:40:52.678 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:03.653150", "end": "2021-09-17 01:40:52.647389", "rc": 0, "start": "2021-09-17 01:40:48.994239", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-09-17T01:40:52.702 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:52 +0000 (0:00:00.040) 0:02:44.711 ****** 2021-09-17T01:40:52.910 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-09-17T01:40:52.912 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-09-17T01:40:52.936 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:52 +0000 (0:00:00.233) 0:02:44.945 ****** 2021-09-17T01:40:52.944 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-09-17T01:40:52.968 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:52 +0000 (0:00:00.032) 0:02:44.978 ****** 2021-09-17T01:40:53.175 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-09-17T01:40:53.178 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T01:40:53.198 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:53 +0000 (0:00:00.229) 0:02:45.207 ****** 2021-09-17T01:40:53.215 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T01:40:53.240 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:53 +0000 (0:00:00.041) 0:02:45.249 ****** 2021-09-17T01:40:53.729 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-09-17T01:40:53.730 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-09-17T01:40:53.772 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-09-17T01:40:54.178 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-09-17T01:40:54.204 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:54 +0000 (0:00:00.963) 0:02:46.213 ****** 2021-09-17T01:40:54.223 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-09-17T01:40:54.248 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:54 +0000 (0:00:00.044) 0:02:46.257 ****** 2021-09-17T01:40:54.748 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi045.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-09-17T01:40:54.791 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-09-17T01:40:55.190 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-09-17T01:40:55.224 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.976) 0:02:47.233 ****** 2021-09-17T01:40:55.273 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-09-17T01:40:55.299 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.074) 0:02:47.308 ****** 2021-09-17T01:40:55.306 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-17T01:40:55.307 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.332 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.032) 0:02:47.340 ****** 2021-09-17T01:40:55.370 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.395 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.063) 0:02:47.404 ****** 2021-09-17T01:40:55.402 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-17T01:40:55.404 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.426 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.031) 0:02:47.436 ****** 2021-09-17T01:40:55.455 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.483 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.054) 0:02:47.490 ****** 2021-09-17T01:40:55.499 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-09-17T01:40:55.501 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.523 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.042) 0:02:47.533 ****** 2021-09-17T01:40:55.551 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.575 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.051) 0:02:47.584 ****** 2021-09-17T01:40:55.593 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-09-17T01:40:55.595 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.617 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.041) 0:02:47.626 ****** 2021-09-17T01:40:55.646 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.669 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.051) 0:02:47.678 ****** 2021-09-17T01:40:55.688 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-09-17T01:40:55.690 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.699 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.701 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.724 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.054) 0:02:47.733 ****** 2021-09-17T01:40:55.741 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.753 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.755 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.781 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.057) 0:02:47.790 ****** 2021-09-17T01:40:55.799 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-17T01:40:55.801 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.824 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.042) 0:02:47.833 ****** 2021-09-17T01:40:55.853 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.877 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.052) 0:02:47.886 ****** 2021-09-17T01:40:55.895 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-09-17T01:40:55.897 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.919 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.042) 0:02:47.928 ****** 2021-09-17T01:40:55.947 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:55.971 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:55 +0000 (0:00:00.051) 0:02:47.980 ****** 2021-09-17T01:40:55.989 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-09-17T01:40:55.990 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:56.012 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:56 +0000 (0:00:00.040) 0:02:48.021 ****** 2021-09-17T01:40:56.040 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:56.067 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:56 +0000 (0:00:00.054) 0:02:48.076 ****** 2021-09-17T01:40:56.085 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-09-17T01:40:56.087 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:56.108 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:56 +0000 (0:00:00.041) 0:02:48.118 ****** 2021-09-17T01:40:56.136 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:56.164 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:56 +0000 (0:00:00.055) 0:02:48.173 ****** 2021-09-17T01:40:56.183 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-09-17T01:40:56.185 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:56.208 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:56 +0000 (0:00:00.043) 0:02:48.217 ****** 2021-09-17T01:40:56.236 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:40:56.238 INFO:teuthology.task.ansible.out: 2021-09-17T01:40:56.261 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:56 +0000 (0:00:00.052) 0:02:48.270 ****** 2021-09-17T01:40:56.517 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-09-17T01:40:56.519 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.063126", "end": "2021-09-17 01:40:56.500732", "rc": 0, "start": "2021-09-17 01:40:56.437606", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.111061", "end": "2021-09-17 01:40:56.500323", "rc": 0, "start": "2021-09-17 01:40:56.389262", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:40:56.541 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:56 +0000 (0:00:00.280) 0:02:48.550 ****** 2021-09-17T01:40:56.561 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:40:56 +0000 (0:00:00.020) 0:02:48.571 ****** 2021-09-17T01:41:11.653 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-09-17T01:41:11.655 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:41:11.676 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:11 +0000 (0:00:15.114) 0:03:03.685 ****** 2021-09-17T01:41:11.756 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-09-17T01:41:11.757 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:11.778 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:11 +0000 (0:00:00.101) 0:03:03.787 ****** 2021-09-17T01:41:11.786 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-09-17T01:41:11.788 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:41:11.811 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:11 +0000 (0:00:00.033) 0:03:03.821 ****** 2021-09-17T01:41:11.933 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 17 September 2021 01:41:11 +0000 (0:00:00.062) 0:03:03.883 ****** TASK [common : Clean apt cache] ************************************************ skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 17 September 2021 01:41:11 +0000 (0:00:00.043) 0:03:03.927 ****** 2021-09-17T01:41:11.945 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-09-17T01:41:11.947 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:11.970 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:11 +0000 (0:00:00.051) 0:03:03.979 ****** 2021-09-17T01:41:11.989 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:12.015 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:12 +0000 (0:00:00.045) 0:03:04.024 ****** 2021-09-17T01:41:12.035 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-09-17T01:41:12.037 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:12.060 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:12 +0000 (0:00:00.044) 0:03:04.069 ****** 2021-09-17T01:41:12.088 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-09-17T01:41:12.089 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:12.111 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:12 +0000 (0:00:00.051) 0:03:04.120 ****** 2021-09-17T01:41:12.180 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:12.202 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:12 +0000 (0:00:00.090) 0:03:04.211 ****** 2021-09-17T01:41:12.572 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-09-17T01:41:12.595 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:12 +0000 (0:00:00.393) 0:03:04.604 ****** 2021-09-17T01:41:12.717 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-09-17T01:41:12.743 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:12 +0000 (0:00:00.147) 0:03:04.752 ****** 2021-09-17T01:41:15.651 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:41:15.674 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:15 +0000 (0:00:02.931) 0:03:07.683 ****** 2021-09-17T01:41:15.954 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:41:15.978 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:15 +0000 (0:00:00.304) 0:03:07.988 ****** 2021-09-17T01:41:16.229 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-09-17T01:41:16.231 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-09-17T01:41:16.520 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-09-17T01:41:16.689 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-09-17T01:41:16.984 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-09-17T01:41:17.145 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-09-17T01:41:17.170 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:17 +0000 (0:00:01.190) 0:03:09.179 ****** 2021-09-17T01:41:17.370 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-09-17T01:41:17.372 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T01:41:17.396 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:17 +0000 (0:00:00.225) 0:03:09.404 ****** 2021-09-17T01:41:17.436 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-09-17T01:41:17.438 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-09-17T01:41:17.463 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:17 +0000 (0:00:00.068) 0:03:09.473 ****** 2021-09-17T01:41:17.670 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T01:41:17.690 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:17 +0000 (0:00:00.226) 0:03:09.700 ****** 2021-09-17T01:41:17.931 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-17T01:41:17.933 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-09-17T01:41:18.194 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-09-17T01:41:18.374 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-09-17T01:41:18.638 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-09-17T01:41:18.818 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-09-17T01:41:18.841 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:18 +0000 (0:00:01.150) 0:03:10.851 ****** 2021-09-17T01:41:18.922 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-09-17T01:41:18.924 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:18.943 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:18 +0000 (0:00:00.101) 0:03:10.953 ****** 2021-09-17T01:41:19.023 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-09-17T01:41:19.025 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-09-17T01:41:19.045 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:19 +0000 (0:00:00.101) 0:03:11.054 ****** 2021-09-17T01:41:19.063 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-17T01:41:19.065 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-09-17T01:41:19.091 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:19 +0000 (0:00:00.045) 0:03:11.100 ****** 2021-09-17T01:41:19.165 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:19.187 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:19 +0000 (0:00:00.096) 0:03:11.196 ****** 2021-09-17T01:41:19.268 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-09-17T01:41:19.269 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-09-17T01:41:19.292 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:19 +0000 (0:00:00.105) 0:03:11.301 ****** 2021-09-17T01:41:19.301 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi013.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-17T01:41:19.327 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:19 +0000 (0:00:00.035) 0:03:11.336 ****** 2021-09-17T01:41:19.421 INFO:teuthology.task.ansible.out: 2021-09-17T01:41:19.428 INFO:teuthology.task.ansible.out:TASK [common : Install nrpe without epel] ************************************** skipping: [smithi013.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:19.434 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:19.445 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:19.466 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:19.490 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:19 +0000 (0:00:00.162) 0:03:11.499 ****** 2021-09-17T01:41:20.543 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-09-17T01:41:20.545 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-17T01:41:20.571 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:20 +0000 (0:00:01.081) 0:03:12.580 ****** 2021-09-17T01:41:20.656 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-09-17T01:41:20.658 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:20.667 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:20.689 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:20.703 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:20.727 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:20 +0000 (0:00:00.156) 0:03:12.737 ****** 2021-09-17T01:41:22.675 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:41:22.700 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:22 +0000 (0:00:01.972) 0:03:14.709 ****** 2021-09-17T01:41:22.773 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:22.795 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:22 +0000 (0:00:00.095) 0:03:14.805 ****** 2021-09-17T01:41:23.316 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-09-17T01:41:23.318 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-09-17T01:41:23.339 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:23 +0000 (0:00:00.543) 0:03:15.348 ****** 2021-09-17T01:41:23.836 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-09-17T01:41:23.838 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-09-17T01:41:23.858 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:23 +0000 (0:00:00.519) 0:03:15.868 ****** 2021-09-17T01:41:23.939 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-09-17T01:41:23.941 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:23.950 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:23.973 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:23 +0000 (0:00:00.114) 0:03:15.982 ****** 2021-09-17T01:41:23.981 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-09-17T01:41:23.982 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:41:24.009 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:24 +0000 (0:00:00.036) 0:03:16.019 ****** 2021-09-17T01:41:24.100 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-09-17T01:41:24.102 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:24.123 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:24 +0000 (0:00:00.113) 0:03:16.132 ****** 2021-09-17T01:41:24.212 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-09-17T01:41:24.214 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T01:41:24.236 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:24 +0000 (0:00:00.113) 0:03:16.245 ****** 2021-09-17T01:41:24.494 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-09-17T01:41:24.496 INFO:teuthology.task.ansible.out:fatal: [smithi013.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-09-17T01:41:24.516 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:24 +0000 (0:00:00.280) 0:03:16.526 ****** 2021-09-17T01:41:24.586 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:24.609 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:24 +0000 (0:00:00.092) 0:03:16.619 ****** 2021-09-17T01:41:24.626 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-09-17T01:41:24.628 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-09-17T01:41:24.653 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:24 +0000 (0:00:00.043) 0:03:16.662 ****** 2021-09-17T01:41:25.125 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-09-17T01:41:25.144 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:25 +0000 (0:00:00.491) 0:03:17.154 ****** 2021-09-17T01:41:25.162 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-09-17T01:41:25.164 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-09-17T01:41:25.190 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:25 +0000 (0:00:00.042) 0:03:17.197 ****** 2021-09-17T01:41:25.269 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi045.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:25.280 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-09-17T01:41:25.304 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:25 +0000 (0:00:00.116) 0:03:17.314 ****** 2021-09-17T01:41:25.558 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-09-17T01:41:25.560 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T01:41:25.588 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:25 +0000 (0:00:00.283) 0:03:17.597 ****** 2021-09-17T01:41:25.826 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-09-17T01:41:25.827 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 01:37:15 UTC", "ActiveEnterTimestampMonotonic": "25041407", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket local-fs.target tmp.mount basic.target systemd-tmpfiles-setup.service time-sync.target system.slice nss-lookup.target network-online.target sysinit.target -.mount var-run.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 01:37:15 UTC", "AssertTimestampMonotonic": "25030422", "Before": "getty@tty1.service shutdown.target multi-user.target plymouth-quit.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 01:37:15 UTC", "ConditionT 2021-09-17T01:41:25.829 INFO:teuthology.task.ansible.out:imestampMonotonic": "25030421", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1229", "ExecMainStartTimestamp": "Fri 2021-09-17 01:37:15 UTC", "ExecMainStartTimestampMonotonic": "25041299", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Fri 2021-09-17 01:37:15 UTC] ; stop_time=[n/a] ; pid=1229 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "985", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 01:37:15 UTC", "InactiveExitTimestampMonotonic": "25041407", "In 2021-09-17T01:41:25.830 INFO:teuthology.task.ansible.out:vocationID": "4031013936e34320b34c4bcefa7c63fd", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1229", "MemoryAccounting": "yes", "MemoryCurrent": "667648", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGr 2021-09-17T01:41:25.832 INFO:teuthology.task.ansible.out:oups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 01:37:15 UTC", "StateChangeTimestampMonotonic": "25041407", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "989", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-09-17 01:37:15 UTC", "WatchdogTimestampMonotonic": "25041406", "WatchdogUSec": "0"}} 2021-09-17T01:41:25.852 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:25 +0000 (0:00:00.264) 0:03:17.861 ****** 2021-09-17T01:41:25.862 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi045.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-09-17T01:41:25.883 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:25 +0000 (0:00:00.030) 0:03:17.892 ****** 2021-09-17T01:41:25.974 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:41:25.997 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:25 +0000 (0:00:00.114) 0:03:18.006 ****** 2021-09-17T01:41:26.089 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-09-17T01:41:26.091 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001875", "end": "2021-09-17 01:41:26.062455", "rc": 0, "start": "2021-09-17 01:41:26.060580", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-09-17T01:41:26.115 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:26 +0000 (0:00:00.117) 0:03:18.124 ****** 2021-09-17T01:41:26.509 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-09-17T01:41:26.511 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-09-17T01:41:26.534 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:26 +0000 (0:00:00.418) 0:03:18.543 ****** 2021-09-17T01:41:26.891 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-09-17T01:41:26.893 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 01:37:15 UTC", "ActiveEnterTimestampMonotonic": "25543714", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount systemd-journald.socket time-sync.target local-fs.target var-run.mount basic.target sysinit.target system.slice systemd-tmpfiles-setup.service -.mount network-online.target nss-lookup.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 01:37:15 UTC", "AssertTimestampMonotonic": "25519503", "Before": "shutdown.target multi-user.target getty@tty1.service plymouth-quit.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 01:37:15 UTC", "ConditionT 2021-09-17T01:41:26.895 INFO:teuthology.task.ansible.out:imestampMonotonic": "25519503", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1223", "ExecMainStartTimestamp": "Fri 2021-09-17 01:37:15 UTC", "ExecMainStartTimestampMonotonic": "25543222", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Fri 2021-09-17 01:37:15 UTC] ; stop_time=[n/a] ; pid=1223 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "985", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 01:37:15 UTC", "InactiveExitTimestampMonotonic": "25543714", "In 2021-09-17T01:41:26.896 INFO:teuthology.task.ansible.out:vocationID": "0387495e5e3a41638b34827c42a934b1", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1223", "MemoryAccounting": "yes", "MemoryCurrent": "667648", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGr 2021-09-17T01:41:26.898 INFO:teuthology.task.ansible.out:oups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 01:37:15 UTC", "StateChangeTimestampMonotonic": "25543714", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "989", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-09-17 01:37:15 UTC", "WatchdogTimestampMonotonic": "25543691", "WatchdogUSec": "0"}} 2021-09-17T01:41:26.918 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:26 +0000 (0:00:00.384) 0:03:18.927 ****** 2021-09-17T01:41:27.149 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-09-17T01:41:27.151 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001829", "end": "2021-09-17 01:41:27.131685", "rc": 0, "start": "2021-09-17 01:41:27.129856", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-09-17T01:41:27.174 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:27 +0000 (0:00:00.254) 0:03:19.182 ****** 2021-09-17T01:41:29.318 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:41:29.339 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:29 +0000 (0:00:02.166) 0:03:21.349 ****** 2021-09-17T01:41:30.407 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:41:30.440 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:41:30 +0000 (0:00:01.100) 0:03:22.449 ****** 2021-09-17T01:42:23.763 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-09-17T01:42:23.765 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-67.el8_4.1.noarch", "Installed: selinux-policy-3.14.3-67.el8_4.1.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch"]} 2021-09-17T01:42:23.787 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:23 +0000 (0:00:53.347) 0:04:15.796 ****** 2021-09-17T01:42:24.579 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-09-17T01:42:24.607 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:24 +0000 (0:00:00.816) 0:04:16.613 ****** 2021-09-17T01:42:25.016 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-09-17T01:42:25.018 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-67.el8_4.1.noarch", "Installed: selinux-policy-3.14.3-67.el8_4.1.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch"]} 2021-09-17T01:42:25.041 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:25 +0000 (0:00:00.435) 0:04:17.049 ****** 2021-09-17T01:42:25.558 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-09-17T01:42:25.607 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:25 +0000 (0:00:00.559) 0:04:17.609 ****** 2021-09-17T01:42:40.740 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.896774", "end": "2021-09-17 01:42:40.715839", "failed_when_result": false, "rc": 0, "start": "2021-09-17 01:42:24.819065", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-09-17T01:42:40.764 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:40 +0000 (0:00:15.163) 0:04:32.773 ****** 2021-09-17T01:42:41.280 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-09-17T01:42:41.282 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-09-17T01:42:41.308 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:41 +0000 (0:00:00.544) 0:04:33.317 ****** 2021-09-17T01:42:41.591 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.027085", "end": "2021-09-17 01:42:41.565851", "rc": 0, "start": "2021-09-17 01:42:41.538766", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:42:41.614 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:41 +0000 (0:00:00.304) 0:04:33.622 ****** 2021-09-17T01:42:41.812 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-09-17T01:42:41.814 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.931709", "end": "2021-09-17 01:42:41.785909", "failed_when_result": false, "rc": 0, "start": "2021-09-17 01:42:25.854200", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-09-17T01:42:41.836 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:41 +0000 (0:00:00.223) 0:04:33.846 ****** 2021-09-17T01:42:41.864 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-09-17T01:42:41.866 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.009806", "end": "2021-09-17 01:42:41.839276", "rc": 0, "start": "2021-09-17 01:42:41.829470", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:42:41.887 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:41 +0000 (0:00:00.050) 0:04:33.896 ****** 2021-09-17T01:42:42.358 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-09-17T01:42:42.360 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-09-17T01:42:42.388 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:42 +0000 (0:00:00.499) 0:04:34.396 ****** 2021-09-17T01:42:42.636 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-09-17T01:42:42.638 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.011055", "end": "2021-09-17 01:42:42.610291", "rc": 0, "start": "2021-09-17 01:42:42.599236", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:42:42.661 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:42 +0000 (0:00:00.273) 0:04:34.670 ****** 2021-09-17T01:42:42.911 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.012852", "end": "2021-09-17 01:42:42.884668", "rc": 0, "start": "2021-09-17 01:42:42.871816", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:42:42.932 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:42 +0000 (0:00:00.271) 0:04:34.942 ****** 2021-09-17T01:42:57.773 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.651203", "end": "2021-09-17 01:42:57.749552", "rc": 0, "start": "2021-09-17 01:42:42.098349", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:42:57.799 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:57 +0000 (0:00:14.865) 0:04:49.807 ****** 2021-09-17T01:42:58.046 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-09-17T01:42:58.048 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-09-17T01:42:58.069 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:58 +0000 (0:00:00.271) 0:04:50.078 ****** 2021-09-17T01:42:58.148 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-09-17T01:42:58.172 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:58 +0000 (0:00:00.098) 0:04:50.177 ****** 2021-09-17T01:42:58.287 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi013.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-17T01:42:58.320 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:58 +0000 (0:00:00.147) 0:04:50.325 ****** 2021-09-17T01:42:58.414 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi013.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "redhat-lsb", "firewalld", "wget", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-09-17T01:42:58.433 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:58 +0000 (0:00:00.113) 0:04:50.439 ****** 2021-09-17T01:42:58.523 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi013.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-17T01:42:58.540 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:58 +0000 (0:00:00.110) 0:04:50.549 ****** 2021-09-17T01:42:58.752 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-17T01:42:58.775 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:58 +0000 (0:00:00.235) 0:04:50.784 ****** 2021-09-17T01:42:58.835 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-09-17T01:42:58.838 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.656913", "end": "2021-09-17 01:42:58.800891", "rc": 0, "start": "2021-09-17 01:42:43.143978", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:42:58.864 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:58 +0000 (0:00:00.086) 0:04:50.871 ****** 2021-09-17T01:42:59.016 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-09-17T01:42:59.018 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-09-17T01:42:59.041 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:59 +0000 (0:00:00.178) 0:04:51.050 ****** 2021-09-17T01:42:59.110 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-09-17T01:42:59.112 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:42:59.134 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:59 +0000 (0:00:00.093) 0:04:51.143 ****** 2021-09-17T01:42:59.143 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-09-17T01:42:59.144 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-09-17T01:42:59.170 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:59 +0000 (0:00:00.036) 0:04:51.180 ****** 2021-09-17T01:42:59.204 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:42:59.231 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:59 +0000 (0:00:00.060) 0:04:51.240 ****** 2021-09-17T01:42:59.258 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-09-17T01:42:59.281 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:59 +0000 (0:00:00.045) 0:04:51.286 ****** 2021-09-17T01:42:59.366 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-17T01:42:59.387 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:59 +0000 (0:00:00.111) 0:04:51.397 ****** 2021-09-17T01:42:59.502 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-09-17T01:42:59.504 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-09-17T01:42:59.540 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:59 +0000 (0:00:00.140) 0:04:51.537 ****** TASK [testnode : Including major version specific variables.] ****************** ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "redhat-lsb", "firewalld", "wget", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-09-17T01:42:59.578 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:59 +0000 (0:00:00.049) 0:04:51.587 ****** 2021-09-17T01:42:59.685 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-17T01:42:59.724 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:59 +0000 (0:00:00.142) 0:04:51.730 ****** 2021-09-17T01:42:59.932 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-17T01:42:59.955 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:42:59 +0000 (0:00:00.233) 0:04:51.963 ****** 2021-09-17T01:43:00.173 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-09-17T01:43:00.175 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-09-17T01:43:00.204 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:00 +0000 (0:00:00.249) 0:04:52.213 ****** 2021-09-17T01:43:00.242 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-09-17T01:43:00.244 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-09-17T01:43:00.272 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:00 +0000 (0:00:00.064) 0:04:52.277 ****** 2021-09-17T01:43:00.295 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi013"}, "changed": false} 2021-09-17T01:43:00.336 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:00 +0000 (0:00:00.062) 0:04:52.340 ****** 2021-09-17T01:43:00.356 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:00.394 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:00 +0000 (0:00:00.063) 0:04:52.403 ****** 2021-09-17T01:43:00.471 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:00.495 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:00 +0000 (0:00:00.100) 0:04:52.504 ****** 2021-09-17T01:43:00.832 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-09-17T01:43:00.834 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-09-17T01:43:00.866 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:00 +0000 (0:00:00.363) 0:04:52.867 ****** 2021-09-17T01:43:01.347 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi013", "ansible_hostname": "smithi013", "ansible_nodename": "smithi013"}, "changed": false, "name": "smithi013"} 2021-09-17T01:43:01.385 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:01 +0000 (0:00:00.526) 0:04:53.394 ****** 2021-09-17T01:43:01.617 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-09-17T01:43:01.645 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:01 +0000 (0:00:00.259) 0:04:53.653 ****** 2021-09-17T01:43:01.793 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi045"}, "changed": false} 2021-09-17T01:43:01.829 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:01 +0000 (0:00:00.184) 0:04:53.837 ****** 2021-09-17T01:43:02.161 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-09-17T01:43:02.187 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:02 +0000 (0:00:00.353) 0:04:54.192 ****** 2021-09-17T01:43:02.376 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-09-17T01:43:02.379 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi045", "ansible_hostname": "smithi045", "ansible_nodename": "smithi045"}, "changed": false, "name": "smithi045"} 2021-09-17T01:43:02.404 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:02 +0000 (0:00:00.219) 0:04:54.411 ****** 2021-09-17T01:43:02.795 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-09-17T01:43:02.832 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:02 +0000 (0:00:00.424) 0:04:54.836 ****** 2021-09-17T01:43:03.006 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-09-17T01:43:03.028 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:03 +0000 (0:00:00.200) 0:04:55.037 ****** 2021-09-17T01:43:03.479 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi013.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-17T01:43:03.499 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:03 +0000 (0:00:00.461) 0:04:55.499 ****** 2021-09-17T01:43:03.622 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-09-17T01:43:03.659 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:03 +0000 (0:00:00.169) 0:04:55.668 ****** 2021-09-17T01:43:04.111 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-09-17T01:43:04.145 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:04 +0000 (0:00:00.485) 0:04:56.154 ****** 2021-09-17T01:43:04.318 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-09-17T01:43:04.320 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-17T01:43:04.351 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:04 +0000 (0:00:00.205) 0:04:56.360 ****** 2021-09-17T01:43:04.401 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-17T01:43:04.441 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:04 +0000 (0:00:00.090) 0:04:56.450 ****** 2021-09-17T01:43:04.707 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-17T01:43:04.709 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-09-17T01:43:04.739 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:04 +0000 (0:00:00.299) 0:04:56.749 ****** 2021-09-17T01:43:04.928 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-09-17T01:43:04.931 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-09-17T01:43:04.956 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:04 +0000 (0:00:00.216) 0:04:56.966 ****** 2021-09-17T01:43:05.024 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* ok: [smithi013.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-09-17T01:43:05.049 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:05 +0000 (0:00:00.092) 0:04:57.058 ****** 2021-09-17T01:43:05.216 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-17T01:43:05.259 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:05 +0000 (0:00:00.207) 0:04:57.266 ****** 2021-09-17T01:43:05.633 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-09-17T01:43:05.642 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:05 +0000 (0:00:00.385) 0:04:57.651 ****** 2021-09-17T01:43:05.740 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-09-17T01:43:05.793 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:05 +0000 (0:00:00.146) 0:04:57.798 ****** 2021-09-17T01:43:05.927 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* ok: [smithi045.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-09-17T01:43:05.957 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:05 +0000 (0:00:00.168) 0:04:57.966 ****** 2021-09-17T01:43:05.959 INFO:teuthology.task.ansible.out: 2021-09-17T01:43:06.459 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-17T01:43:06.464 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-09-17T01:43:06.499 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:06 +0000 (0:00:00.541) 0:04:58.507 ****** 2021-09-17T01:43:06.582 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-17T01:43:06.585 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-09-17T01:43:06.608 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:06 +0000 (0:00:00.109) 0:04:58.617 ****** 2021-09-17T01:43:07.166 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-09-17T01:43:07.191 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:07 +0000 (0:00:00.583) 0:04:59.201 ****** 2021-09-17T01:43:08.965 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-09-17T01:43:08.966 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.227991", "end": "2021-09-17 01:43:08.939061", "rc": 0, "start": "2021-09-17 01:43:06.711070", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:43:08.990 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:08 +0000 (0:00:01.797) 0:05:00.998 ****** 2021-09-17T01:43:09.088 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:09.112 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:09 +0000 (0:00:00.117) 0:05:01.116 ****** 2021-09-17T01:43:09.199 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T01:43:09.201 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:09.224 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:09 +0000 (0:00:00.118) 0:05:01.234 ****** 2021-09-17T01:43:09.328 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* skipping: [smithi013.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-17T01:43:09.350 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:09 +0000 (0:00:00.125) 0:05:01.360 ****** 2021-09-17T01:43:09.447 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:09.466 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:09 +0000 (0:00:00.115) 0:05:01.475 ****** 2021-09-17T01:43:09.612 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-09-17T01:43:09.613 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.161943", "end": "2021-09-17 01:43:09.581278", "rc": 0, "start": "2021-09-17 01:43:07.419335", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:43:09.643 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:09 +0000 (0:00:00.176) 0:05:01.652 ****** 2021-09-17T01:43:09.756 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-09-17T01:43:09.757 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:09.788 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:09 +0000 (0:00:00.145) 0:05:01.798 ****** 2021-09-17T01:43:09.882 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:09.905 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:09 +0000 (0:00:00.116) 0:05:01.914 ****** 2021-09-17T01:43:09.996 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-09-17T01:43:09.998 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-17T01:43:10.019 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:10 +0000 (0:00:00.113) 0:05:02.028 ****** 2021-09-17T01:43:10.082 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* ok: [smithi013.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-09-17T01:43:10.124 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:10 +0000 (0:00:00.099) 0:05:02.127 ****** 2021-09-17T01:43:10.130 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-09-17T01:43:10.138 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:10.160 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:10 +0000 (0:00:00.041) 0:05:02.169 ****** 2021-09-17T01:43:10.214 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-09-17T01:43:10.243 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:10 +0000 (0:00:00.083) 0:05:02.252 ****** 2021-09-17T01:43:10.774 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-09-17T01:43:10.776 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-09-17T01:43:10.800 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:10 +0000 (0:00:00.555) 0:05:02.808 ****** 2021-09-17T01:43:10.870 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-09-17T01:43:10.897 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:10 +0000 (0:00:00.096) 0:05:02.905 ****** 2021-09-17T01:43:11.213 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-09-17T01:43:11.214 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.704486", "end": "2021-09-17 01:43:11.190183", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-09-17 01:43:10.485697", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Updating Subscription Management repositories.\nRepository successfully enabled.", "stdout_lines": ["Updating Subscription Management repositories.", "Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use 2021-09-17T01:43:11.216 INFO:teuthology.task.ansible.out: command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-09-17T01:43:11.237 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:11 +0000 (0:00:00.341) 0:05:03.246 ****** 2021-09-17T01:43:11.308 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-09-17T01:43:11.310 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:11.332 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:11 +0000 (0:00:00.094) 0:05:03.341 ****** 2021-09-17T01:43:11.426 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-09-17T01:43:11.428 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-17T01:43:11.451 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:11 +0000 (0:00:00.118) 0:05:03.459 ****** 2021-09-17T01:43:11.532 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-09-17T01:43:11.535 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:11.567 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:11 +0000 (0:00:00.117) 0:05:03.576 ****** 2021-09-17T01:43:11.648 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:11.671 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:11 +0000 (0:00:00.103) 0:05:03.680 ****** 2021-09-17T01:43:11.753 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:11.780 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:11 +0000 (0:00:00.109) 0:05:03.789 ****** 2021-09-17T01:43:11.853 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-09-17T01:43:11.855 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.702678", "end": "2021-09-17 01:43:11.825516", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-09-17 01:43:11.122838", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Updating Subscription Management repositories.\nRepository successfully enabled.", "stdout_lines": ["Updating Subscription Management repositories.", "Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-09-17T01:43:11.882 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:11 +0000 (0:00:00.102) 0:05:03.891 ****** 2021-09-17T01:43:11.899 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-09-17T01:43:11.901 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:11.923 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:11 +0000 (0:00:00.040) 0:05:03.932 ****** 2021-09-17T01:43:11.966 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:11.991 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:11 +0000 (0:00:00.068) 0:05:04.000 ****** 2021-09-17T01:43:12.080 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi045.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-17T01:43:12.101 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:12 +0000 (0:00:00.110) 0:05:04.111 ****** 2021-09-17T01:43:12.104 INFO:teuthology.task.ansible.out: 2021-09-17T01:43:12.200 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-09-17T01:43:12.202 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:12.232 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:12 +0000 (0:00:00.130) 0:05:04.241 ****** 2021-09-17T01:43:12.240 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-09-17T01:43:12.242 INFO:teuthology.task.ansible.out:fatal: [smithi013.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007459", "end": "2021-09-17 01:43:12.201624", "msg": "non-zero return code", "rc": 1, "start": "2021-09-17 01:43:12.194165", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-09-17T01:43:12.268 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:12 +0000 (0:00:00.036) 0:05:04.277 ****** 2021-09-17T01:43:12.312 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:12.338 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:12 +0000 (0:00:00.069) 0:05:04.347 ****** 2021-09-17T01:43:12.358 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-09-17T01:43:12.360 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:12.389 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:12 +0000 (0:00:00.050) 0:05:04.398 ****** 2021-09-17T01:43:12.409 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-09-17T01:43:12.411 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:12.442 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:12 +0000 (0:00:00.052) 0:05:04.450 ****** 2021-09-17T01:43:12.522 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-09-17T01:43:12.524 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:12.549 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:12 +0000 (0:00:00.106) 0:05:04.556 ****** 2021-09-17T01:43:12.816 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* fatal: [smithi045.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007449", "end": "2021-09-17 01:43:12.784709", "msg": "non-zero return code", "rc": 1, "start": "2021-09-17 01:43:12.777260", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-09-17T01:43:12.830 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:12 +0000 (0:00:00.282) 0:05:04.839 ****** 2021-09-17T01:43:12.904 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:43:12.927 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:12 +0000 (0:00:00.097) 0:05:04.937 ****** 2021-09-17T01:43:43.573 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-17T01:43:43.575 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-17T01:43:43.597 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:43 +0000 (0:00:30.668) 0:05:35.606 ****** 2021-09-17T01:43:43.945 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-17T01:43:43.975 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:43:43 +0000 (0:00:00.378) 0:05:35.984 ****** 2021-09-17T01:44:12.810 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-09-17T01:44:12.836 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:44:12 +0000 (0:00:28.860) 0:06:04.844 ****** 2021-09-17T01:44:13.174 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-09-17T01:44:13.210 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:44:13 +0000 (0:00:00.367) 0:06:05.212 ****** 2021-09-17T01:44:46.341 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-img-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-17T01:44:46.370 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:44:46 +0000 (0:00:33.164) 0:06:38.377 ****** 2021-09-17T01:44:46.760 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-img-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-17T01:44:46.784 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:44:46 +0000 (0:00:00.416) 0:06:38.793 ****** 2021-09-17T01:45:01.062 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:45:01.087 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:01 +0000 (0:00:14.303) 0:06:53.096 ****** 2021-09-17T01:45:01.318 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:45:01.340 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:01 +0000 (0:00:00.253) 0:06:53.349 ****** 2021-09-17T01:45:04.535 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-17T01:45:04.537 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:45:04.559 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:04 +0000 (0:00:03.217) 0:06:56.567 ****** 2021-09-17T01:45:04.642 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-17T01:45:04.643 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:04.667 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:04 +0000 (0:00:00.107) 0:06:56.674 ****** 2021-09-17T01:45:04.794 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:45:04.830 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:04 +0000 (0:00:00.162) 0:06:56.837 ****** 2021-09-17T01:45:04.910 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-17T01:45:04.912 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:04.933 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:04 +0000 (0:00:00.105) 0:06:56.943 ****** 2021-09-17T01:45:05.032 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket system.slice sysinit.target basic.target dbus.service polkit.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service ebtables.service nftables.service ip6tables.service iptables.service shutdown.target", "ControlPID": "0", "DefaultDepend 2021-09-17T01:45:05.034 INFO:teuthology.task.ansible.out:encies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft 2021-09-17T01:45:05.035 INFO:teuthology.task.ansible.out:": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKI 2021-09-17T01:45:05.038 INFO:teuthology.task.ansible.out:LL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-17T01:45:05.061 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:05 +0000 (0:00:00.126) 0:06:57.069 ****** 2021-09-17T01:45:05.303 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket dbus.service polkit.service sysinit.target basic.target system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service ip6tables.service iptables.service ebtables.service nftables.service shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", 2021-09-17T01:45:05.305 INFO:teuthology.task.ansible.out:"Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": " 2021-09-17T01:45:05.306 INFO:teuthology.task.ansible.out:0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null 2021-09-17T01:45:05.308 INFO:teuthology.task.ansible.out:", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-17T01:45:05.327 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:05 +0000 (0:00:00.266) 0:06:57.336 ****** 2021-09-17T01:45:05.397 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-09-17T01:45:05.399 INFO:teuthology.task.ansible.out:fatal: [smithi013.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-09-17T01:45:05.424 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:05 +0000 (0:00:00.097) 0:06:57.433 ****** 2021-09-17T01:45:05.664 INFO:teuthology.task.ansible.out:fatal: [smithi045.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-09-17T01:45:05.688 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:05 +0000 (0:00:00.264) 0:06:57.698 ****** 2021-09-17T01:45:06.083 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-09-17T01:45:06.085 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-09-17T01:45:06.106 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.417) 0:06:58.115 ****** 2021-09-17T01:45:06.113 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-09-17T01:45:06.142 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.036) 0:06:58.151 ****** 2021-09-17T01:45:06.182 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.207 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.064) 0:06:58.216 ****** 2021-09-17T01:45:06.225 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.249 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.042) 0:06:58.259 ****** 2021-09-17T01:45:06.281 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.304 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.054) 0:06:58.313 ****** 2021-09-17T01:45:06.325 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.350 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.045) 0:06:58.359 ****** 2021-09-17T01:45:06.377 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-09-17T01:45:06.379 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.403 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.053) 0:06:58.413 ****** 2021-09-17T01:45:06.432 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.465 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.061) 0:06:58.474 ****** 2021-09-17T01:45:06.495 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.519 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.053) 0:06:58.528 ****** 2021-09-17T01:45:06.549 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.572 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.053) 0:06:58.582 ****** 2021-09-17T01:45:06.601 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-09-17T01:45:06.603 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.633 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.054) 0:06:58.636 ****** 2021-09-17T01:45:06.647 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.675 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.047) 0:06:58.684 ****** 2021-09-17T01:45:06.720 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.762 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.084) 0:06:58.769 ****** 2021-09-17T01:45:06.770 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:06.798 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.037) 0:06:58.807 ****** 2021-09-17T01:45:06.837 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-09-17T01:45:06.878 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.080) 0:06:58.887 ****** 2021-09-17T01:45:06.954 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:06 +0000 (0:00:00.075) 0:06:58.963 ****** 2021-09-17T01:45:06.997 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-09-17T01:45:06.999 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.021 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.067) 0:06:59.031 ****** 2021-09-17T01:45:07.042 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.075 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.053) 0:06:59.085 ****** 2021-09-17T01:45:07.107 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-09-17T01:45:07.109 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.131 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.055) 0:06:59.141 ****** 2021-09-17T01:45:07.150 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.174 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.042) 0:06:59.183 ****** 2021-09-17T01:45:07.204 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi013.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.206 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.229 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.055) 0:06:59.238 ****** 2021-09-17T01:45:07.247 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.259 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.292 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.062) 0:06:59.301 ****** 2021-09-17T01:45:07.309 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-09-17T01:45:07.311 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.343 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.050) 0:06:59.352 ****** 2021-09-17T01:45:07.369 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.393 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.050) 0:06:59.402 ****** 2021-09-17T01:45:07.422 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-09-17T01:45:07.448 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.054) 0:06:59.457 ****** 2021-09-17T01:45:07.490 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.042) 0:06:59.499 ****** 2021-09-17T01:45:07.521 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.545 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.055) 0:06:59.555 ****** 2021-09-17T01:45:07.565 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.588 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.042) 0:06:59.597 ****** 2021-09-17T01:45:07.617 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.643 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.055) 0:06:59.653 ****** 2021-09-17T01:45:07.661 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.688 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.043) 0:06:59.696 ****** 2021-09-17T01:45:07.716 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-09-17T01:45:07.718 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.741 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.054) 0:06:59.750 ****** 2021-09-17T01:45:07.771 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.795 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.054) 0:06:59.804 ****** 2021-09-17T01:45:07.824 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-17T01:45:07.827 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.853 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.057) 0:06:59.862 ****** 2021-09-17T01:45:07.873 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.898 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.046) 0:06:59.908 ****** 2021-09-17T01:45:07.926 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-09-17T01:45:07.928 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:07.953 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:07 +0000 (0:00:00.054) 0:06:59.962 ****** 2021-09-17T01:45:07.983 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.009 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.056) 0:07:00.018 ****** 2021-09-17T01:45:08.028 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-17T01:45:08.030 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.054 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.043) 0:07:00.062 ****** 2021-09-17T01:45:08.083 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.106 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.053) 0:07:00.115 ****** 2021-09-17T01:45:08.123 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-09-17T01:45:08.125 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.151 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.045) 0:07:00.160 ****** 2021-09-17T01:45:08.180 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.207 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.056) 0:07:00.217 ****** 2021-09-17T01:45:08.225 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-09-17T01:45:08.227 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.258 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.046) 0:07:00.263 ****** 2021-09-17T01:45:08.288 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.312 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.058) 0:07:00.322 ****** 2021-09-17T01:45:08.350 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-09-17T01:45:08.352 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.381 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.068) 0:07:00.390 ****** 2021-09-17T01:45:08.390 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.417 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.035) 0:07:00.426 ****** 2021-09-17T01:45:08.452 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.476 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.059) 0:07:00.485 ****** 2021-09-17T01:45:08.487 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.511 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.035) 0:07:00.520 ****** 2021-09-17T01:45:08.549 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-09-17T01:45:08.551 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.574 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.062) 0:07:00.583 ****** 2021-09-17T01:45:08.591 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.617 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.043) 0:07:00.626 ****** 2021-09-17T01:45:08.638 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.666 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.048) 0:07:00.674 ****** 2021-09-17T01:45:08.695 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.724 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.058) 0:07:00.733 ****** 2021-09-17T01:45:08.742 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-09-17T01:45:08.745 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.773 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.042) 0:07:00.776 ****** 2021-09-17T01:45:08.808 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.810 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:08.838 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.070) 0:07:00.846 ****** 2021-09-17T01:45:08.866 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.890 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.052) 0:07:00.899 ****** 2021-09-17T01:45:08.917 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.949 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.058) 0:07:00.958 ****** 2021-09-17T01:45:08.976 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-09-17T01:45:08.978 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:08.999 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:08 +0000 (0:00:00.050) 0:07:01.008 ****** 2021-09-17T01:45:09.017 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.041 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.041) 0:07:01.050 ****** 2021-09-17T01:45:09.069 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-17T01:45:09.071 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.092 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.051) 0:07:01.101 ****** 2021-09-17T01:45:09.121 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.149 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.056) 0:07:01.158 ****** 2021-09-17T01:45:09.151 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:09.180 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.206 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.057) 0:07:01.216 ****** 2021-09-17T01:45:09.228 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.255 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.047) 0:07:01.263 ****** 2021-09-17T01:45:09.282 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-09-17T01:45:09.284 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.306 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.052) 0:07:01.316 ****** 2021-09-17T01:45:09.334 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.363 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.056) 0:07:01.372 ****** 2021-09-17T01:45:09.381 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-17T01:45:09.383 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.406 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.042) 0:07:01.415 ****** 2021-09-17T01:45:09.433 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.461 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.054) 0:07:01.470 ****** 2021-09-17T01:45:09.480 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.516 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.055) 0:07:01.525 ****** 2021-09-17T01:45:09.536 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.565 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.049) 0:07:01.575 ****** 2021-09-17T01:45:09.568 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:09.600 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.623 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.057) 0:07:01.632 ****** 2021-09-17T01:45:09.653 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.654 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:09.688 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.066) 0:07:01.698 ****** 2021-09-17T01:45:09.710 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.735 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.046) 0:07:01.744 ****** 2021-09-17T01:45:09.753 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.778 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.043) 0:07:01.787 ****** 2021-09-17T01:45:09.807 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-17T01:45:09.809 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.832 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.053) 0:07:01.841 ****** 2021-09-17T01:45:09.851 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.877 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.044) 0:07:01.886 ****** 2021-09-17T01:45:09.908 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.932 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.056) 0:07:01.942 ****** 2021-09-17T01:45:09.951 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:09.982 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:09 +0000 (0:00:00.049) 0:07:01.991 ****** 2021-09-17T01:45:10.014 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-17T01:45:10.016 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.043 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.060) 0:07:02.052 ****** 2021-09-17T01:45:10.064 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.097 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.054) 0:07:02.106 ****** 2021-09-17T01:45:10.118 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.150 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.052) 0:07:02.159 ****** 2021-09-17T01:45:10.181 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.206 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.055) 0:07:02.215 ****** 2021-09-17T01:45:10.242 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-17T01:45:10.244 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.268 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.063) 0:07:02.278 ****** 2021-09-17T01:45:10.270 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:10.291 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.326 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.057) 0:07:02.336 ****** 2021-09-17T01:45:10.349 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.376 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.049) 0:07:02.385 ****** 2021-09-17T01:45:10.407 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.431 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.054) 0:07:02.440 ****** 2021-09-17T01:45:10.448 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-17T01:45:10.450 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.473 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.041) 0:07:02.482 ****** 2021-09-17T01:45:10.514 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.541 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.068) 0:07:02.550 ****** 2021-09-17T01:45:10.560 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-17T01:45:10.562 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.585 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.043) 0:07:02.594 ****** 2021-09-17T01:45:10.614 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.640 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.054) 0:07:02.649 ****** 2021-09-17T01:45:10.680 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.704 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.063) 0:07:02.713 ****** 2021-09-17T01:45:10.716 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.739 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.035) 0:07:02.749 ****** 2021-09-17T01:45:10.768 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.792 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.052) 0:07:02.801 ****** 2021-09-17T01:45:10.811 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.835 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.042) 0:07:02.844 ****** 2021-09-17T01:45:10.863 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-17T01:45:10.865 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.888 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.052) 0:07:02.897 ****** 2021-09-17T01:45:10.906 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.932 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.043) 0:07:02.941 ****** 2021-09-17T01:45:10.960 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-17T01:45:10.962 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:10.986 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:10 +0000 (0:00:00.054) 0:07:02.996 ****** 2021-09-17T01:45:11.005 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.031 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.044) 0:07:03.041 ****** 2021-09-17T01:45:11.059 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-17T01:45:11.061 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.086 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.054) 0:07:03.095 ****** 2021-09-17T01:45:11.105 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.107 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:11.131 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.045) 0:07:03.141 ****** 2021-09-17T01:45:11.164 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.189 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.058) 0:07:03.199 ****** 2021-09-17T01:45:11.219 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.245 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.055) 0:07:03.254 ****** 2021-09-17T01:45:11.273 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-17T01:45:11.276 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.298 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.052) 0:07:03.307 ****** 2021-09-17T01:45:11.329 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.366 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.067) 0:07:03.375 ****** 2021-09-17T01:45:11.385 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.409 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.043) 0:07:03.419 ****** 2021-09-17T01:45:11.450 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.475 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.065) 0:07:03.484 ****** 2021-09-17T01:45:11.497 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-17T01:45:11.498 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.523 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.048) 0:07:03.533 ****** 2021-09-17T01:45:11.553 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.578 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.053) 0:07:03.587 ****** 2021-09-17T01:45:11.618 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-17T01:45:11.620 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.656 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.076) 0:07:03.663 ****** 2021-09-17T01:45:11.675 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.701 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.046) 0:07:03.710 ****** 2021-09-17T01:45:11.725 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-09-17T01:45:11.728 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.766 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.065) 0:07:03.776 ****** 2021-09-17T01:45:11.789 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.821 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.054) 0:07:03.830 ****** 2021-09-17T01:45:11.850 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-09-17T01:45:11.852 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.874 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.053) 0:07:03.884 ****** 2021-09-17T01:45:11.894 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.922 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.046) 0:07:03.930 ****** 2021-09-17T01:45:11.952 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:11.979 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:11 +0000 (0:00:00.058) 0:07:03.988 ****** 2021-09-17T01:45:11.997 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.021 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.041) 0:07:04.030 ****** 2021-09-17T01:45:12.049 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-17T01:45:12.050 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.074 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.053) 0:07:04.084 ****** 2021-09-17T01:45:12.094 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.122 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.047) 0:07:04.132 ****** 2021-09-17T01:45:12.152 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.182 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.056) 0:07:04.188 ****** 2021-09-17T01:45:12.216 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.243 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.063) 0:07:04.252 ****** 2021-09-17T01:45:12.263 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.294 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.051) 0:07:04.304 ****** 2021-09-17T01:45:12.330 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.354 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.059) 0:07:04.363 ****** 2021-09-17T01:45:12.355 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:12.384 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-09-17T01:45:12.387 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.411 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.057) 0:07:04.420 ****** 2021-09-17T01:45:12.431 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.459 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.046) 0:07:04.467 ****** 2021-09-17T01:45:12.486 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.510 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.052) 0:07:04.520 ****** 2021-09-17T01:45:12.542 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.566 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.055) 0:07:04.575 ****** 2021-09-17T01:45:12.595 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-09-17T01:45:12.596 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.622 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.055) 0:07:04.631 ****** 2021-09-17T01:45:12.641 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.667 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.044) 0:07:04.676 ****** 2021-09-17T01:45:12.708 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.731 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.064) 0:07:04.740 ****** 2021-09-17T01:45:12.749 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.775 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.044) 0:07:04.785 ****** 2021-09-17T01:45:12.807 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.833 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.057) 0:07:04.842 ****** 2021-09-17T01:45:12.852 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.875 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.042) 0:07:04.884 ****** 2021-09-17T01:45:12.905 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-09-17T01:45:12.907 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.933 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.058) 0:07:04.943 ****** 2021-09-17T01:45:12.953 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:12.979 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:12 +0000 (0:00:00.045) 0:07:04.989 ****** 2021-09-17T01:45:13.011 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-09-17T01:45:13.013 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.048 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.055) 0:07:05.044 ****** 2021-09-17T01:45:13.056 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.083 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.047) 0:07:05.091 ****** 2021-09-17T01:45:13.122 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-09-17T01:45:13.145 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.062) 0:07:05.154 ****** 2021-09-17T01:45:13.164 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-09-17T01:45:13.189 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.043) 0:07:05.198 ****** 2021-09-17T01:45:13.236 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi013.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} ok: [smithi013.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-17T01:45:13.246 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.258 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.262 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.270 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.271 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.300 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.110) 0:07:05.309 ****** 2021-09-17T01:45:13.309 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-17T01:45:13.311 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.311 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.311 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.323 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.325 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.362 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.054) 0:07:05.364 ****** 2021-09-17T01:45:13.577 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-09-17T01:45:13.578 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.008626", "end": "2021-09-17 01:45:13.552344", "rc": 0, "start": "2021-09-17 01:45:13.543718", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_63133ca92c299bd23f40f94d370cf5381b0b1313/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_63133ca92c299bd23f40f94d370cf5381b0b1313/virtualenv/bin/teuthology-lock"]} 2021-09-17T01:45:13.606 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.251) 0:07:05.615 ****** 2021-09-17T01:45:13.627 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004582", "end": "2021-09-17 01:45:13.598056", "rc": 0, "start": "2021-09-17 01:45:13.593474", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_63133ca92c299bd23f40f94d370cf5381b0b1313/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_63133ca92c299bd23f40f94d370cf5381b0b1313/virtualenv/bin/teuthology-lock"]} 2021-09-17T01:45:13.656 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.049) 0:07:05.664 ****** 2021-09-17T01:45:13.685 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.713 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.057) 0:07:05.722 ****** 2021-09-17T01:45:13.731 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.759 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.046) 0:07:05.768 ****** 2021-09-17T01:45:13.761 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:13.803 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-09-17T01:45:13.805 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.829 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.069) 0:07:05.838 ****** 2021-09-17T01:45:13.847 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.874 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.045) 0:07:05.883 ****** 2021-09-17T01:45:13.904 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-09-17T01:45:13.906 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.931 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.056) 0:07:05.940 ****** 2021-09-17T01:45:13.961 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:13.986 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:13 +0000 (0:00:00.054) 0:07:05.995 ****** 2021-09-17T01:45:14.004 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-09-17T01:45:14.006 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:14.029 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:14 +0000 (0:00:00.043) 0:07:06.039 ****** 2021-09-17T01:45:14.060 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:14.086 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:14 +0000 (0:00:00.055) 0:07:06.094 ****** 2021-09-17T01:45:17.231 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:45:17.254 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:17 +0000 (0:00:03.169) 0:07:09.264 ****** 2021-09-17T01:45:17.328 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:45:17.353 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:17 +0000 (0:00:00.098) 0:07:09.362 ****** 2021-09-17T01:45:17.366 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi013.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320625664, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129059, 'block_used': 12966049, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129059, "block_size": 4096, "block_total": 240095108, "block_used": 12966049, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320625664, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}} 2021-09-17T01:45:17.397 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:17 +0000 (0:00:00.043) 0:07:09.406 ****** 2021-09-17T01:45:17.453 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320625664, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129059, 'block_used': 12966049, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129059, "block_size": 4096, "block_total": 240095108, "block_used": 12966049, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320625664, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}} 2021-09-17T01:45:17.482 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:17 +0000 (0:00:00.083) 0:07:09.490 ****** 2021-09-17T01:45:17.686 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005377", "end": "2021-09-17 01:45:17.654826", "rc": 0, "start": "2021-09-17 01:45:17.649449", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:17.708 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:17 +0000 (0:00:00.227) 0:07:09.717 ****** 2021-09-17T01:45:17.759 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005416", "end": "2021-09-17 01:45:17.720313", "rc": 0, "start": "2021-09-17 01:45:17.714897", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:17.795 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:17 +0000 (0:00:00.086) 0:07:09.804 ****** 2021-09-17T01:45:17.815 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi013.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320625664, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129059, 'block_used': 12966049, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129059, "block_size": 4096, "block_total": 240095108, "block_used": 12966049, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320625664, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:17.845 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:17 +0000 (0:00:00.044) 0:07:09.848 ****** 2021-09-17T01:45:17.904 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320625664, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129059, 'block_used': 12966049, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129059, "block_size": 4096, "block_total": 240095108, "block_used": 12966049, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320625664, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:17.945 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:17 +0000 (0:00:00.105) 0:07:09.954 ****** 2021-09-17T01:45:17.974 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-09-17T01:45:18.002 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:17 +0000 (0:00:00.049) 0:07:10.003 ****** 2021-09-17T01:45:18.048 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:18 +0000 (0:00:00.053) 0:07:10.056 ****** 2021-09-17T01:45:19.446 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAT0A0GetU9U2Zem5IplqjDfOW4dzUHayI'], 'uuids': ['d919dab5-be2e-4de1-94c2-ca6c96e56525'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.153252", "end": "2021-09-17 01:45:19.407125", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAT0A0GetU9U2Zem5IplqjDfOW4dzUHayI"], "labels": [], "masters": [], "uuids": ["d919dab5-be2e-4de1-94c2-ca6c96e56525"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:18.253873", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk u 2021-09-17T01:45:19.448 INFO:teuthology.task.ansible.out:sing fdisk or", "other utilities."]} 2021-09-17T01:45:19.490 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfLOCF5mYfMugMtb4OpbxW6rWiahu85JHc'], 'uuids': ['f041d038-9ae9-46c0-a124-a494dd25894c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.140442", "end": "2021-09-17 01:45:19.458124", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfLOCF5mYfMugMtb4OpbxW6rWiahu85JHc"], "labels": [], "masters": [], "uuids": ["f041d038-9ae9-46c0-a124-a494dd25894c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:18.317682", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T01:45:20.696 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-QyyIns-3hbh-Qckq-W0Mv-RLZR-2OxJ-p61GPq', 'nvme-INTEL_SSDPED1D480GA_PHMB7476008K480DGN', 'nvme-nvme.8086-50484d42373437363030384b34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.045981", "end": "2021-09-17 01:45:20.665070", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-QyyIns-3hbh-Qckq-W0Mv-RLZR-2OxJ-p61GPq", "nvme-INTEL_SSDPED1D480GA_PHMB7476008K480DGN", "nvme-nvme.8086-50484d42373437363030384b34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:19.619089", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-09-17T01:45:20.697 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T01:45:20.748 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-g12S6h-lqie-8l2f-tTev-21Gd-B8XW-GImwi9', 'nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN', 'nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.058284", "end": "2021-09-17 01:45:20.718456", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-g12S6h-lqie-8l2f-tTev-21Gd-B8XW-GImwi9", "nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN", "nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:19.660172", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-09-17T01:45:20.750 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T01:45:21.945 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAYkWIjgCGJ5ZfPuB8dzHg14fOgjHK3LGM'], 'uuids': ['eea27ad4-1d6d-4cff-8bae-b83b212e3fd4'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.050191", "end": "2021-09-17 01:45:21.916506", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAYkWIjgCGJ5ZfPuB8dzHg14fOgjHK3LGM"], "labels": [], "masters": [], "uuids": ["eea27ad4-1d6d-4cff-8bae-b83b212e3fd4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:20.866315", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T01:45:21.998 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfmysXpXhJ50dKiC2pxOItV3ca8lvJN3kO'], 'uuids': ['2cb4dbae-b165-416d-81f7-1085736ef12b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.050340", "end": "2021-09-17 01:45:21.968097", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfmysXpXhJ50dKiC2pxOItV3ca8lvJN3kO"], "labels": [], "masters": [], "uuids": ["2cb4dbae-b165-416d-81f7-1085736ef12b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:20.917757", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T01:45:23.198 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyA9aenaciSeOgnCCQniJ5GeMl6cChRQTnp'], 'uuids': ['7fd69e46-0978-4c35-bcfa-8063a5096200'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.045102", "end": "2021-09-17 01:45:23.166057", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyA9aenaciSeOgnCCQniJ5GeMl6cChRQTnp"], "labels": [], "masters": [], "uuids": ["7fd69e46-0978-4c35-bcfa-8063a5096200"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:22.120955", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T01:45:23.250 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfDL668p2YNrCsrz3xOxCfmbUVzRdksbN6'], 'uuids': ['95845ab2-fa13-4e10-800e-52096fd0fe4d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.047658", "end": "2021-09-17 01:45:23.218033", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfDL668p2YNrCsrz3xOxCfmbUVzRdksbN6"], "labels": [], "masters": [], "uuids": ["95845ab2-fa13-4e10-800e-52096fd0fe4d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:22.170375", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T01:45:24.443 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAY2pzsC24f9WjjPm1KRzZNhzNzR6Ic6Ly'], 'uuids': ['5c53268a-76fb-4638-a6a9-b5d6d2f0a7fe'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.046132", "end": "2021-09-17 01:45:24.407957", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAY2pzsC24f9WjjPm1KRzZNhzNzR6Ic6Ly"], "labels": [], "masters": [], "uuids": ["5c53268a-76fb-4638-a6a9-b5d6d2f0a7fe"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:23.361825", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T01:45:24.455 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2YD', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4J2YD', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J2YD', 'scsi-35000c500871b846a', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4J2YD', 'wwn-0x5000c500871b846a'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4J2YD', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871b846a', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2YD-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4J2YD-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J2YD-part1', 'scsi-35000c500871b846a-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4J2YD-part1', 'wwn-0x5000c500871b846a-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2YD", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4J2YD", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J2YD", "scsi-35000c500871b846a", "scsi-SATA_ST1000NM0033-9ZM_Z1W4J2YD", "wwn-0x5000c500871b846a"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2YD-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4J2YD-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-17T01:45:24.457 INFO:teuthology.task.ansible.out:Z1W4J2YD-part1", "scsi-35000c500871b846a-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4J2YD-part1", "wwn-0x5000c500871b846a-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2YD", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b846a"}}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:24.521 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfy3m0pJXTT3H0VdW62t8cbn1buJ0iflTw'], 'uuids': ['879d60e3-650b-4f33-80d5-62fdc4c29420'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.072151", "end": "2021-09-17 01:45:24.486691", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfy3m0pJXTT3H0VdW62t8cbn1buJ0iflTw"], "labels": [], "masters": [], "uuids": ["879d60e3-650b-4f33-80d5-62fdc4c29420"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:23.414540", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T01:45:24.533 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPNL', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL', 'scsi-35000c500871a9bab', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL', 'wwn-0x5000c500871a9bab'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HPNL', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a9bab', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL-part1', 'scsi-35000c500871a9bab-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL-part1', 'wwn-0x5000c500871a9bab-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL", "scsi-35000c500871a9bab", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL", "wwn-0x5000c500871a9bab"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-17T01:45:24.535 INFO:teuthology.task.ansible.out:Z1W4HPNL-part1", "scsi-35000c500871a9bab-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL-part1", "wwn-0x5000c500871a9bab-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPNL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a9bab"}}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:25.702 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAnFtIY9jrNtTQ9d9v0puwS7IIQ11CXQjy'], 'uuids': ['c3089ca9-726b-4893-aa9b-d1e3e576ccdb'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.053421", "end": "2021-09-17 01:45:25.667894", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAnFtIY9jrNtTQ9d9v0puwS7IIQ11CXQjy"], "labels": [], "masters": [], "uuids": ["c3089ca9-726b-4893-aa9b-d1e3e576ccdb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:24.614473", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T01:45:25.732 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:25 +0000 (0:00:07.684) 0:07:17.741 ****** 2021-09-17T01:45:25.867 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfayBawWDGBSdkTCX9s1ZQRZb4JSBiqVlg'], 'uuids': ['ac3f2c7d-c0df-408b-9cc9-3f72c7aadd05'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.145182", "end": "2021-09-17 01:45:25.836794", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfayBawWDGBSdkTCX9s1ZQRZb4JSBiqVlg"], "labels": [], "masters": [], "uuids": ["ac3f2c7d-c0df-408b-9cc9-3f72c7aadd05"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:24.691612", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T01:45:25.910 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:25 +0000 (0:00:00.178) 0:07:17.919 ****** 2021-09-17T01:45:26.105 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-17T01:45:26.107 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAT0A0GetU9U2Zem5IplqjDfOW4dzUHayI'], 'uuids': ['d919dab5-be2e-4de1-94c2-ca6c96e56525'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.101812", "end": "2021-09-17 01:45:26.074713", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAT0A0GetU9U2Zem5IplqjDfOW4dzUHayI"], "labels": [], "masters": [], "uuids": ["d919dab5-be2e-4de1-94c2-ca6c96e56525"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:25.972901", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0989246 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0989246 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:26.285 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfLOCF5mYfMugMtb4OpbxW6rWiahu85JHc'], 'uuids': ['f041d038-9ae9-46c0-a124-a494dd25894c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.100764", "end": "2021-09-17 01:45:26.251659", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfLOCF5mYfMugMtb4OpbxW6rWiahu85JHc"], "labels": [], "masters": [], "uuids": ["f041d038-9ae9-46c0-a124-a494dd25894c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:26.150895", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0979852 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0979852 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:26.318 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-QyyIns-3hbh-Qckq-W0Mv-RLZR-2OxJ-p61GPq', 'nvme-INTEL_SSDPED1D480GA_PHMB7476008K480DGN', 'nvme-nvme.8086-50484d42373437363030384b34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039611", "end": "2021-09-17 01:45:26.294849", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-QyyIns-3hbh-Qckq-W0Mv-RLZR-2OxJ-p61GPq", "nvme-INTEL_SSDPED1D480GA_PHMB7476008K480DGN", "nvme-nvme.8086-50484d42373437363030384b34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:26.255238", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0367421 s, 3.1 GB/s", "stderr_lines": [" 2021-09-17T01:45:26.320 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0367421 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:26.504 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-g12S6h-lqie-8l2f-tTev-21Gd-B8XW-GImwi9', 'nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN', 'nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039392", "end": "2021-09-17 01:45:26.475791", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-g12S6h-lqie-8l2f-tTev-21Gd-B8XW-GImwi9", "nvme-INTEL_SSDPED1D480GA_PHMB75120098480DGN", "nvme-nvme.8086-50484d42373531323030393834383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:26.436399", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0365526 s, 3.2 GB/s", "stderr_lines": [" 2021-09-17T01:45:26.506 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0365526 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:26.602 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAYkWIjgCGJ5ZfPuB8dzHg14fOgjHK3LGM'], 'uuids': ['eea27ad4-1d6d-4cff-8bae-b83b212e3fd4'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.100070", "end": "2021-09-17 01:45:26.572776", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAYkWIjgCGJ5ZfPuB8dzHg14fOgjHK3LGM"], "labels": [], "masters": [], "uuids": ["eea27ad4-1d6d-4cff-8bae-b83b212e3fd4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:26.472706", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0972901 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0972901 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:26.781 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfmysXpXhJ50dKiC2pxOItV3ca8lvJN3kO'], 'uuids': ['2cb4dbae-b165-416d-81f7-1085736ef12b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.100047", "end": "2021-09-17 01:45:26.759872", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfmysXpXhJ50dKiC2pxOItV3ca8lvJN3kO"], "labels": [], "masters": [], "uuids": ["2cb4dbae-b165-416d-81f7-1085736ef12b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:26.659825", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.097286 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.097286 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:26.888 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyA9aenaciSeOgnCCQniJ5GeMl6cChRQTnp'], 'uuids': ['7fd69e46-0978-4c35-bcfa-8063a5096200'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.101325", "end": "2021-09-17 01:45:26.856065", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyA9aenaciSeOgnCCQniJ5GeMl6cChRQTnp"], "labels": [], "masters": [], "uuids": ["7fd69e46-0978-4c35-bcfa-8063a5096200"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:26.754740", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0985504 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0985504 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:27.067 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfDL668p2YNrCsrz3xOxCfmbUVzRdksbN6'], 'uuids': ['95845ab2-fa13-4e10-800e-52096fd0fe4d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.100601", "end": "2021-09-17 01:45:27.038434", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfDL668p2YNrCsrz3xOxCfmbUVzRdksbN6"], "labels": [], "masters": [], "uuids": ["95845ab2-fa13-4e10-800e-52096fd0fe4d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:26.937833", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0978226 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0978226 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:27.168 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAY2pzsC24f9WjjPm1KRzZNhzNzR6Ic6Ly'], 'uuids': ['5c53268a-76fb-4638-a6a9-b5d6d2f0a7fe'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.099765", "end": "2021-09-17 01:45:27.140505", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAY2pzsC24f9WjjPm1KRzZNhzNzR6Ic6Ly"], "labels": [], "masters": [], "uuids": ["5c53268a-76fb-4638-a6a9-b5d6d2f0a7fe"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:27.040740", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0969499 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0969499 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:27.180 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2YD', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4J2YD', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J2YD', 'scsi-35000c500871b846a', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4J2YD', 'wwn-0x5000c500871b846a'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4J2YD', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871b846a', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2YD-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4J2YD-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J2YD-part1', 'scsi-35000c500871b846a-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4J2YD-part1', 'wwn-0x5000c500871b846a-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2YD", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4J2YD", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4J2YD", "scsi-35000c500871b846a", "scsi-SATA_ST1000NM0033-9ZM_Z1W4J2YD", "wwn-0x5000c500871b846a"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2YD-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4J2YD-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-17T01:45:27.182 INFO:teuthology.task.ansible.out:Z1W4J2YD-part1", "scsi-35000c500871b846a-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4J2YD-part1", "wwn-0x5000c500871b846a-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2YD", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b846a"}}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:27.339 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfy3m0pJXTT3H0VdW62t8cbn1buJ0iflTw'], 'uuids': ['879d60e3-650b-4f33-80d5-62fdc4c29420'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.100441", "end": "2021-09-17 01:45:27.317994", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfy3m0pJXTT3H0VdW62t8cbn1buJ0iflTw"], "labels": [], "masters": [], "uuids": ["879d60e3-650b-4f33-80d5-62fdc4c29420"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:27.217553", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0976052 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0976052 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:27.352 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPNL', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL', 'scsi-35000c500871a9bab', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL', 'wwn-0x5000c500871a9bab'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HPNL', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a9bab', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL-part1', 'scsi-35000c500871a9bab-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL-part1', 'wwn-0x5000c500871a9bab-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPNL", "scsi-35000c500871a9bab", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL", "wwn-0x5000c500871a9bab"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPNL-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPNL-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-17T01:45:27.353 INFO:teuthology.task.ansible.out:Z1W4HPNL-part1", "scsi-35000c500871a9bab-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPNL-part1", "wwn-0x5000c500871a9bab-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPNL", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a9bab"}}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:27.457 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAnFtIY9jrNtTQ9d9v0puwS7IIQ11CXQjy'], 'uuids': ['c3089ca9-726b-4893-aa9b-d1e3e576ccdb'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.100312", "end": "2021-09-17 01:45:27.428057", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-iaAUM8HklVl5KyCcoPRNYWSN1nMDzIyAnFtIY9jrNtTQ9d9v0puwS7IIQ11CXQjy"], "labels": [], "masters": [], "uuids": ["c3089ca9-726b-4893-aa9b-d1e3e576ccdb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:27.327745", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0975934 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0975934 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:27.491 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:27 +0000 (0:00:01.576) 0:07:19.496 ****** 2021-09-17T01:45:27.632 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfayBawWDGBSdkTCX9s1ZQRZb4JSBiqVlg'], 'uuids': ['ac3f2c7d-c0df-408b-9cc9-3f72c7aadd05'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.100618", "end": "2021-09-17 01:45:27.604937", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Ag8w2GhXJJyAsPXuPdBdS4JDpQKtGCGfayBawWDGBSdkTCX9s1ZQRZb4JSBiqVlg"], "labels": [], "masters": [], "uuids": ["ac3f2c7d-c0df-408b-9cc9-3f72c7aadd05"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 01:45:27.504319", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0977407 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0977407 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:27.662 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:27 +0000 (0:00:00.175) 0:07:19.672 ****** 2021-09-17T01:45:27.798 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-09-17T01:45:27.801 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.070864", "end": "2021-09-17 01:45:27.725817", "rc": 0, "start": "2021-09-17 01:45:27.654953", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:27.823 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:27 +0000 (0:00:00.160) 0:07:19.832 ****** 2021-09-17T01:45:27.967 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.066172", "end": "2021-09-17 01:45:27.894050", "rc": 0, "start": "2021-09-17 01:45:27.827878", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:27.994 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:27 +0000 (0:00:00.169) 0:07:20.002 ****** 2021-09-17T01:45:28.065 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-09-17T01:45:28.067 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.048265", "end": "2021-09-17 01:45:28.039938", "rc": 0, "start": "2021-09-17 01:45:27.991673", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:28.092 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.099) 0:07:20.101 ****** 2021-09-17T01:45:28.162 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-09-17T01:45:28.244 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.152) 0:07:20.253 ****** 2021-09-17T01:45:28.252 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-09-17T01:45:28.253 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.051678", "end": "2021-09-17 01:45:28.214386", "rc": 0, "start": "2021-09-17 01:45:28.162708", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:28.278 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.033) 0:07:20.287 ****** 2021-09-17T01:45:28.286 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-09-17T01:45:28.287 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.315 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.037) 0:07:20.324 ****** 2021-09-17T01:45:28.322 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-09-17T01:45:28.350 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.035) 0:07:20.360 ****** 2021-09-17T01:45:28.357 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.382 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.032) 0:07:20.392 ****** 2021-09-17T01:45:28.389 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-09-17T01:45:28.391 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.415 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.032) 0:07:20.424 ****** 2021-09-17T01:45:28.422 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-17T01:45:28.424 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.457 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.040) 0:07:20.465 ****** 2021-09-17T01:45:28.470 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-09-17T01:45:28.471 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.496 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.040) 0:07:20.505 ****** 2021-09-17T01:45:28.503 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-09-17T01:45:28.504 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.528 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.031) 0:07:20.537 ****** 2021-09-17T01:45:28.535 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-17T01:45:28.537 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.564 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.036) 0:07:20.574 ****** 2021-09-17T01:45:28.596 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-09-17T01:45:28.598 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.622 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.057) 0:07:20.631 ****** 2021-09-17T01:45:28.739 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi013.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T01:45:28.761 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.139) 0:07:20.770 ****** 2021-09-17T01:45:28.803 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi013.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320625664, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129059, 'block_used': 12966049, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129059, "block_size": 4096, "block_total": 240095108, "block_used": 12966049, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320625664, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.826 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.065) 0:07:20.836 ****** 2021-09-17T01:45:28.834 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-09-17T01:45:28.836 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T01:45:28.859 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.032) 0:07:20.868 ****** 2021-09-17T01:45:28.866 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-09-17T01:45:28.868 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.893 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.033) 0:07:20.901 ****** 2021-09-17T01:45:28.910 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-17T01:45:28.912 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320625664, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129059, 'block_used': 12966049, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129059, "block_size": 4096, "block_total": 240095108, "block_used": 12966049, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320625664, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.935 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.042) 0:07:20.944 ****** 2021-09-17T01:45:28.943 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:28.967 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.032) 0:07:20.976 ****** 2021-09-17T01:45:28.974 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:29.004 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:28 +0000 (0:00:00.035) 0:07:21.012 ****** 2021-09-17T01:45:29.021 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:29.050 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:29 +0000 (0:00:00.046) 0:07:21.059 ****** 2021-09-17T01:45:29.058 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T01:45:29.060 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:29.083 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:29 +0000 (0:00:00.033) 0:07:21.092 ****** 2021-09-17T01:45:29.090 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-09-17T01:45:29.094 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:29.119 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:29 +0000 (0:00:00.035) 0:07:21.128 ****** 2021-09-17T01:45:29.126 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T01:45:29.128 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:29.154 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:29 +0000 (0:00:00.034) 0:07:21.163 ****** 2021-09-17T01:45:29.171 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-09-17T01:45:29.173 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:29.194 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:29 +0000 (0:00:00.040) 0:07:21.204 ****** 2021-09-17T01:45:29.201 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-09-17T01:45:29.203 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:29.228 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:29 +0000 (0:00:00.033) 0:07:21.237 ****** 2021-09-17T01:45:29.237 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:29.271 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:29 +0000 (0:00:00.043) 0:07:21.280 ****** 2021-09-17T01:45:29.285 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-09-17T01:45:29.287 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:29.315 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:29 +0000 (0:00:00.041) 0:07:21.322 ****** 2021-09-17T01:45:29.342 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-09-17T01:45:29.344 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:29.368 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:29 +0000 (0:00:00.055) 0:07:21.377 ****** 2021-09-17T01:45:29.783 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-09-17T01:45:29.787 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-09-17T01:45:29.807 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:29 +0000 (0:00:00.438) 0:07:21.816 ****** 2021-09-17T01:45:29.835 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-09-17T01:45:29.854 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:29 +0000 (0:00:00.047) 0:07:21.863 ****** 2021-09-17T01:45:30.459 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-17T01:45:30.462 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T01:45:30.803 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T01:45:30.824 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T01:45:31.133 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T01:45:31.164 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T01:45:31.481 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T01:45:31.513 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T01:45:31.824 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T01:45:31.850 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:31 +0000 (0:00:01.995) 0:07:23.859 ****** 2021-09-17T01:45:31.870 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T01:45:31.898 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:31 +0000 (0:00:00.048) 0:07:23.907 ****** 2021-09-17T01:45:32.052 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-17T01:45:32.054 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-17T01:45:32.077 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:32 +0000 (0:00:00.178) 0:07:24.086 ****** 2021-09-17T01:45:32.108 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-17T01:45:32.133 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:32 +0000 (0:00:00.055) 0:07:24.142 ****** 2021-09-17T01:45:32.298 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-17T01:45:32.300 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T01:45:32.353 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T01:45:32.475 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T01:45:32.550 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T01:45:32.656 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T01:45:32.721 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T01:45:32.837 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T01:45:32.850 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:32.878 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:32 +0000 (0:00:00.744) 0:07:24.887 ****** 2021-09-17T01:45:32.906 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-09-17T01:45:32.908 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T01:45:32.931 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:32 +0000 (0:00:00.053) 0:07:24.940 ****** 2021-09-17T01:45:32.939 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:32.968 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:32 +0000 (0:00:00.035) 0:07:24.976 ****** 2021-09-17T01:45:32.973 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-09-17T01:45:32.975 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:32.999 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:32 +0000 (0:00:00.032) 0:07:25.008 ****** 2021-09-17T01:45:33.008 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:33.031 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:33 +0000 (0:00:00.031) 0:07:25.040 ****** 2021-09-17T01:45:33.068 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-09-17T01:45:33.070 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:33.091 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:33 +0000 (0:00:00.060) 0:07:25.100 ****** 2021-09-17T01:45:33.206 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T01:45:33.232 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:33 +0000 (0:00:00.139) 0:07:25.240 ****** 2021-09-17T01:45:33.260 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-09-17T01:45:33.262 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:33.291 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:33 +0000 (0:00:00.058) 0:07:25.298 ****** 2021-09-17T01:45:33.309 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-09-17T01:45:33.311 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T01:45:33.335 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:33 +0000 (0:00:00.046) 0:07:25.344 ****** 2021-09-17T01:45:33.375 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-09-17T01:45:33.377 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:33.401 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:33 +0000 (0:00:00.064) 0:07:25.409 ****** 2021-09-17T01:45:33.777 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-09-17T01:45:33.779 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true} 2021-09-17T01:45:33.801 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:33 +0000 (0:00:00.401) 0:07:25.810 ****** 2021-09-17T01:45:33.810 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true} 2021-09-17T01:45:33.834 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:33 +0000 (0:00:00.033) 0:07:25.844 ****** 2021-09-17T01:45:34.361 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-09-17T01:45:34.366 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-09-17T01:45:34.383 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.549) 0:07:26.393 ****** 2021-09-17T01:45:34.396 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-09-17T01:45:34.398 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:34.424 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.040) 0:07:26.433 ****** 2021-09-17T01:45:34.432 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-09-17T01:45:34.433 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.457 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.032) 0:07:26.466 ****** 2021-09-17T01:45:34.466 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.489 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.032) 0:07:26.498 ****** 2021-09-17T01:45:34.496 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-09-17T01:45:34.497 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.522 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.031) 0:07:26.530 ****** 2021-09-17T01:45:34.533 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.558 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.036) 0:07:26.567 ****** 2021-09-17T01:45:34.565 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-17T01:45:34.567 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.591 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.032) 0:07:26.600 ****** 2021-09-17T01:45:34.600 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.624 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.033) 0:07:26.633 ****** 2021-09-17T01:45:34.632 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-17T01:45:34.634 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.658 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.033) 0:07:26.667 ****** 2021-09-17T01:45:34.667 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.693 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.035) 0:07:26.702 ****** 2021-09-17T01:45:34.700 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-09-17T01:45:34.702 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.728 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.035) 0:07:26.738 ****** 2021-09-17T01:45:34.737 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.763 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.033) 0:07:26.771 ****** 2021-09-17T01:45:34.770 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:34.772 INFO:teuthology.task.ansible.out:TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.797 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.034) 0:07:26.806 ****** 2021-09-17T01:45:34.805 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.830 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.033) 0:07:26.839 ****** 2021-09-17T01:45:34.838 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-09-17T01:45:34.839 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.865 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.034) 0:07:26.874 ****** 2021-09-17T01:45:34.875 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.898 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.033) 0:07:26.907 ****** 2021-09-17T01:45:34.900 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:34.906 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-09-17T01:45:34.908 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.932 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.033) 0:07:26.941 ****** 2021-09-17T01:45:34.940 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:34.965 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.032) 0:07:26.974 ****** 2021-09-17T01:45:34.973 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-09-17T01:45:34.974 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.000 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:34 +0000 (0:00:00.035) 0:07:27.009 ****** 2021-09-17T01:45:35.009 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.037 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.036) 0:07:27.046 ****** 2021-09-17T01:45:35.046 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.069 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.032) 0:07:27.078 ****** 2021-09-17T01:45:35.078 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.107 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.037) 0:07:27.116 ****** 2021-09-17T01:45:35.114 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-09-17T01:45:35.118 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.145 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.036) 0:07:27.153 ****** 2021-09-17T01:45:35.159 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.185 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.040) 0:07:27.194 ****** 2021-09-17T01:45:35.194 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.219 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.034) 0:07:27.228 ****** 2021-09-17T01:45:35.226 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.253 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.034) 0:07:27.262 ****** 2021-09-17T01:45:35.260 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-09-17T01:45:35.262 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.285 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.030) 0:07:27.293 ****** 2021-09-17T01:45:35.292 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.320 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.032) 0:07:27.326 ****** 2021-09-17T01:45:35.324 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.349 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.032) 0:07:27.358 ****** 2021-09-17T01:45:35.358 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.388 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.039) 0:07:27.397 ****** 2021-09-17T01:45:35.395 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.427 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.039) 0:07:27.436 ****** 2021-09-17T01:45:35.437 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:35.473 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.042) 0:07:27.479 ****** 2021-09-17T01:45:35.933 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-09-17T01:45:35.934 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-09-17T01:45:35.963 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:35 +0000 (0:00:00.493) 0:07:27.973 ****** 2021-09-17T01:45:35.982 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-09-17T01:45:36.007 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.043) 0:07:28.016 ****** 2021-09-17T01:45:36.273 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-09-17T01:45:36.274 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 01:37:08 UTC", "ActiveEnterTimestampMonotonic": "17932274", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket tmp.mount sysinit.target ntpdate.service sntp.service -.mount basic.target ntpd.service systemd-tmpfiles-setup.service system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 01:37:07 UTC", "AssertTimestampMonotonic": "16567196", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 01:37:07 UTC", "ConditionTimestampMonotonic": "16567132", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service 2021-09-17T01:45:36.276 INFO:teuthology.task.ansible.out: shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "870", "ExecMainStartTimestamp": "Fri 2021-09-17 01:37:08 UTC", "ExecMainStartTimestampMonotonic": "17852555", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 01:37:07 UTC", "InactiveExitTimestampMonotonic": "16570812", "InvocationID": "9af34badbb79441f9ce591b434b89515", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinit 2021-09-17T01:45:36.277 INFO:teuthology.task.ansible.out:y", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "870", "MemoryAccounting": "yes", "MemoryCurrent": "1257472", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit. 2021-09-17T01:45:36.279 INFO:teuthology.task.ansible.out:target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 01:37:08 UTC", "StateChangeTimestampMonotonic": "17932274", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-09-17 01:37:08 UTC", "WatchdogTimestampMonotonic": "17852598", "WatchdogUSec": "0"}} 2021-09-17T01:45:36.299 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.291) 0:07:28.308 ****** 2021-09-17T01:45:36.319 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 01:37:08 UTC", "ActiveEnterTimestampMonotonic": "18456347", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target -.mount ntpdate.service systemd-journald.socket ntpd.service systemd-tmpfiles-setup.service sysinit.target sntp.service system.slice tmp.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 01:37:06 UTC", "AssertTimestampMonotonic": "17074642", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 01:37:06 UTC", "ConditionTimestampMonotonic": "17073190", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.tar 2021-09-17T01:45:36.321 INFO:teuthology.task.ansible.out:get ntpd.service systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "883", "ExecMainStartTimestamp": "Fri 2021-09-17 01:37:07 UTC", "ExecMainStartTimestampMonotonic": "18338814", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 01:37:06 UTC", "InactiveExitTimestampMonotonic": "17096255", "InvocationID": "b32df15f15e74505bf97cec5f91f10f7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinit 2021-09-17T01:45:36.323 INFO:teuthology.task.ansible.out:y", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "883", "MemoryAccounting": "yes", "MemoryCurrent": "1249280", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system 2021-09-17T01:45:36.325 INFO:teuthology.task.ansible.out:.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 01:37:08 UTC", "StateChangeTimestampMonotonic": "18456347", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-09-17 01:37:07 UTC", "WatchdogTimestampMonotonic": "18338838", "WatchdogUSec": "0"}} 2021-09-17T01:45:36.343 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.044) 0:07:28.353 ****** 2021-09-17T01:45:36.350 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T01:45:36.352 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-09-17T01:45:36.377 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.033) 0:07:28.386 ****** 2021-09-17T01:45:36.384 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-09-17T01:45:36.409 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.032) 0:07:28.418 ****** 2021-09-17T01:45:36.426 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T01:45:36.428 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-17T01:45:36.450 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.041) 0:07:28.459 ****** 2021-09-17T01:45:36.458 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-17T01:45:36.487 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.037) 0:07:28.496 ****** 2021-09-17T01:45:36.493 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-09-17T01:45:36.495 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:36.522 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.035) 0:07:28.531 ****** 2021-09-17T01:45:36.530 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:36.554 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.032) 0:07:28.563 ****** 2021-09-17T01:45:36.562 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-09-17T01:45:36.564 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:36.592 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.037) 0:07:28.601 ****** 2021-09-17T01:45:36.600 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:36.624 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.032) 0:07:28.633 ****** 2021-09-17T01:45:36.632 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:36.656 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.031) 0:07:28.665 ****** 2021-09-17T01:45:36.665 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:36.688 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.032) 0:07:28.697 ****** 2021-09-17T01:45:36.859 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-09-17T01:45:36.882 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.194) 0:07:28.892 ****** 2021-09-17T01:45:36.904 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-09-17T01:45:36.927 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:36 +0000 (0:00:00.044) 0:07:28.936 ****** 2021-09-17T01:45:37.375 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-09-17T01:45:37.379 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-09-17T01:45:37.402 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:37 +0000 (0:00:00.474) 0:07:29.411 ****** 2021-09-17T01:45:37.422 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-09-17T01:45:37.447 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:37 +0000 (0:00:00.044) 0:07:29.455 ****** 2021-09-17T01:45:38.706 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-09-17T01:45:38.736 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:38 +0000 (0:00:01.289) 0:07:30.745 ****** 2021-09-17T01:45:38.744 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-09-17T01:45:38.770 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:38 +0000 (0:00:00.033) 0:07:30.779 ****** 2021-09-17T01:45:38.778 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:38.805 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:38 +0000 (0:00:00.035) 0:07:30.814 ****** 2021-09-17T01:45:38.824 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:38.855 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:38 +0000 (0:00:00.050) 0:07:30.865 ****** 2021-09-17T01:45:38.863 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-09-17T01:45:38.865 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-09-17T01:45:38.891 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:38 +0000 (0:00:00.035) 0:07:30.900 ****** 2021-09-17T01:45:38.899 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-09-17T01:45:38.923 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:38 +0000 (0:00:00.032) 0:07:30.933 ****** 2021-09-17T01:45:39.079 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-09-17T01:45:39.081 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-09-17T01:45:39.105 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.181) 0:07:31.114 ****** 2021-09-17T01:45:39.124 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-09-17T01:45:39.157 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.047) 0:07:31.162 ****** 2021-09-17T01:45:39.170 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-09-17T01:45:39.173 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:39.197 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.044) 0:07:31.206 ****** 2021-09-17T01:45:39.206 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:39.234 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.036) 0:07:31.243 ****** 2021-09-17T01:45:39.247 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-17T01:45:39.273 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.038) 0:07:31.281 ****** 2021-09-17T01:45:39.279 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-17T01:45:39.305 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.033) 0:07:31.314 ****** 2021-09-17T01:45:39.313 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-09-17T01:45:39.315 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:39.338 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.031) 0:07:31.346 ****** 2021-09-17T01:45:39.346 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:39.369 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.032) 0:07:31.378 ****** 2021-09-17T01:45:39.533 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-09-17T01:45:39.535 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-17T01:45:39.558 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.189) 0:07:31.567 ****** 2021-09-17T01:45:39.578 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-17T01:45:39.602 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.044) 0:07:31.612 ****** 2021-09-17T01:45:39.759 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-09-17T01:45:39.761 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-09-17T01:45:39.786 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.182) 0:07:31.794 ****** 2021-09-17T01:45:39.806 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-09-17T01:45:39.831 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.046) 0:07:31.840 ****** 2021-09-17T01:45:39.839 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-09-17T01:45:39.841 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:39.867 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.035) 0:07:31.875 ****** 2021-09-17T01:45:39.873 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:39.902 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.035) 0:07:31.911 ****** 2021-09-17T01:45:39.921 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-09-17T01:45:39.922 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:39.923 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:39.935 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:39.937 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:39.966 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.064) 0:07:31.975 ****** 2021-09-17T01:45:39.974 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:39.976 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.005 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:39 +0000 (0:00:00.038) 0:07:32.014 ****** 2021-09-17T01:45:40.017 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi045.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.019 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.032 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi045.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.042 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.073 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.067) 0:07:32.081 ****** 2021-09-17T01:45:40.085 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi013.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.094 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.097 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.118 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.046) 0:07:32.128 ****** 2021-09-17T01:45:40.126 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T01:45:40.128 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.155 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.035) 0:07:32.164 ****** 2021-09-17T01:45:40.161 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.189 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.034) 0:07:32.199 ****** 2021-09-17T01:45:40.198 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-09-17T01:45:40.199 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.224 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.033) 0:07:32.232 ****** 2021-09-17T01:45:40.240 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.256 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.033) 0:07:32.265 ****** 2021-09-17T01:45:40.264 INFO:teuthology.task.ansible.out: 2021-09-17T01:45:40.266 INFO:teuthology.task.ansible.out:TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.292 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.036) 0:07:32.302 ****** 2021-09-17T01:45:40.301 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.330 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.035) 0:07:32.337 ****** 2021-09-17T01:45:40.335 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-09-17T01:45:40.337 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.360 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.032) 0:07:32.370 ****** 2021-09-17T01:45:40.367 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.392 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.031) 0:07:32.401 ****** 2021-09-17T01:45:40.400 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.427 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.034) 0:07:32.435 ****** 2021-09-17T01:45:40.434 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.463 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.034) 0:07:32.470 ****** 2021-09-17T01:45:40.477 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.499 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.038) 0:07:32.508 ****** 2021-09-17T01:45:40.507 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.538 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.035) 0:07:32.544 ****** 2021-09-17T01:45:40.543 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-09-17T01:45:40.545 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.571 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.035) 0:07:32.579 ****** 2021-09-17T01:45:40.578 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.605 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.034) 0:07:32.614 ****** 2021-09-17T01:45:40.613 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-17T01:45:40.615 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.638 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.033) 0:07:32.647 ****** 2021-09-17T01:45:40.647 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:40.675 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:40 +0000 (0:00:00.036) 0:07:32.684 ****** 2021-09-17T01:45:40.875 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi045.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.006133", "end": "2021-09-17 01:45:40.825972", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-17 01:45:40.819839", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:40.922 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005915", "end": "2021-09-17 01:45:40.881008", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-17 01:45:40.875093", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:41.053 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003239", "end": "2021-09-17 01:45:41.019959", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-17 01:45:41.016720", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:41.109 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003214", "end": "2021-09-17 01:45:41.067315", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-17 01:45:41.064101", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:41.236 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003010", "end": "2021-09-17 01:45:41.196974", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-17 01:45:41.193964", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:41.415 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003545", "end": "2021-09-17 01:45:41.383215", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-17 01:45:41.379670", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:41.440 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:41 +0000 (0:00:00.764) 0:07:33.449 ****** 2021-09-17T01:45:41.468 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-09-17T01:45:41.501 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:41 +0000 (0:00:00.056) 0:07:33.506 ****** 2021-09-17T01:45:41.536 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-09-17T01:45:41.538 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:42.297 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-09-17T01:45:42.299 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:01.004180", "end": "2021-09-17 01:45:42.258607", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-17 01:45:41.254427", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:42.491 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003569", "end": "2021-09-17 01:45:42.453874", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-17 01:45:42.450305", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:45:42.514 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:42 +0000 (0:00:01.018) 0:07:34.524 ****** 2021-09-17T01:45:42.542 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-09-17T01:45:42.565 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:42 +0000 (0:00:00.050) 0:07:34.574 ****** 2021-09-17T01:45:42.593 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:42.661 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-09-17T01:45:42.684 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:42 +0000 (0:00:00.118) 0:07:34.693 ****** 2021-09-17T01:45:42.706 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:42 +0000 (0:00:00.021) 0:07:34.714 ****** 2021-09-17T01:45:43.362 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-09-17T01:45:43.364 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] 2021-09-17T01:45:43.403 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] 2021-09-17T01:45:43.433 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:43 +0000 (0:00:00.727) 0:07:35.442 ****** 2021-09-17T01:45:43.478 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:43 +0000 (0:00:00.045) 0:07:35.488 ****** 2021-09-17T01:45:43.486 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:43.511 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:43 +0000 (0:00:00.032) 0:07:35.520 ****** 2021-09-17T01:45:43.519 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:45:43.545 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:43 +0000 (0:00:00.033) 0:07:35.554 ****** 2021-09-17T01:45:43.554 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-09-17T01:45:43.580 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:43 +0000 (0:00:00.034) 0:07:35.589 ****** 2021-09-17T01:45:43.587 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-09-17T01:45:43.614 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:43 +0000 (0:00:00.034) 0:07:35.623 ****** 2021-09-17T01:45:43.644 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-09-17T01:45:43.647 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-09-17T01:45:43.670 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:43 +0000 (0:00:00.055) 0:07:35.678 ****** 2021-09-17T01:45:43.688 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-09-17T01:45:43.718 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:45:43 +0000 (0:00:00.043) 0:07:35.722 ****** 2021-09-17T01:46:04.521 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-09-17T01:46:04.523 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-docker-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.noarch", "Installed: podman-catatonit-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Installed: podman-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Removed: podman-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-catatonit-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-docker-3.0.1-6.module+el8.4.0+10607+f4da7515.noarch"]} 2021-09-17T01:46:04.547 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:04 +0000 (0:00:20.834) 0:07:56.556 ****** 2021-09-17T01:46:04.573 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-docker-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.noarch", "Installed: podman-catatonit-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Installed: podman-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Removed: podman-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-catatonit-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-docker-3.0.1-6.module+el8.4.0+10607+f4da7515.noarch"]} 2021-09-17T01:46:04.596 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:04 +0000 (0:00:00.049) 0:07:56.605 ****** 2021-09-17T01:46:04.608 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi013.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-09-17T01:46:04.631 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:04 +0000 (0:00:00.034) 0:07:56.640 ****** 2021-09-17T01:46:04.642 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-09-17T01:46:04.671 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:04 +0000 (0:00:00.038) 0:07:56.679 ****** 2021-09-17T01:46:04.680 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-17T01:46:04.682 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:46:04.730 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:04 +0000 (0:00:00.057) 0:07:56.737 ****** 2021-09-17T01:46:04.738 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:46:04.773 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:04 +0000 (0:00:00.039) 0:07:56.776 ****** 2021-09-17T01:46:04.972 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* ok: [smithi013.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T01:46:05.017 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T01:46:05.168 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T01:46:05.196 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:05 +0000 (0:00:00.429) 0:07:57.206 ****** 2021-09-17T01:46:05.221 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T01:46:05.247 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:05 +0000 (0:00:00.049) 0:07:57.256 ****** 2021-09-17T01:46:05.738 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-09-17T01:46:05.740 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-17T01:46:05.880 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-17T01:46:06.196 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-17T01:46:06.221 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:06 +0000 (0:00:00.975) 0:07:58.231 ****** 2021-09-17T01:46:06.363 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-17T01:46:06.388 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:06 +0000 (0:00:00.165) 0:07:58.396 ****** 2021-09-17T01:46:20.635 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-09-17T01:46:20.638 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:46:20.660 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:20 +0000 (0:00:14.272) 0:08:12.669 ****** 2021-09-17T01:46:21.021 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-17T01:46:21.046 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:21 +0000 (0:00:00.385) 0:08:13.055 ****** 2021-09-17T01:46:23.220 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** ok: [smithi013.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-61601dmw-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-61601dmw-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0 2021-09-17T01:46:23.221 INFO:teuthology.task.ansible.out::", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-17T01:46:23.250 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:23 +0000 (0:00:02.204) 0:08:15.260 ****** 2021-09-17T01:46:23.278 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:46:23.303 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:23 +0000 (0:00:00.052) 0:08:15.312 ****** 2021-09-17T01:46:23.341 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-09-17T01:46:23.343 INFO:teuthology.task.ansible.out:skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:46:23.371 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:23 +0000 (0:00:00.067) 0:08:15.380 ****** 2021-09-17T01:46:23.634 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-09-17T01:46:23.636 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-cykg5rsv-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-cykg5rsv-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py in 2021-09-17T01:46:23.637 INFO:teuthology.task.ansible.out:stall for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-17T01:46:23.660 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:23 +0000 (0:00:00.288) 0:08:15.668 ****** 2021-09-17T01:46:23.698 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:46:23.720 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:23 +0000 (0:00:00.061) 0:08:15.729 ****** 2021-09-17T01:46:23.737 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-09-17T01:46:23.739 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.158530", "end": "2021-09-17 01:46:23.710794", "rc": 0, "start": "2021-09-17 01:46:23.552264", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:46:23.766 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:23 +0000 (0:00:00.046) 0:08:15.776 ****** 2021-09-17T01:46:23.774 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-09-17T01:46:23.774 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:46:23.799 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:23 +0000 (0:00:00.032) 0:08:15.808 ****** 2021-09-17T01:46:23.807 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* skipping: [smithi013.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:46:24.168 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-09-17T01:46:24.170 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.167982", "end": "2021-09-17 01:46:24.151977", "rc": 0, "start": "2021-09-17 01:46:23.983995", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T01:46:24.190 INFO:teuthology.task.ansible.out:Friday 17 September 2021 01:46:24 +0000 (0:00:00.391) 0:08:16.199 ****** 2021-09-17T01:46:24.218 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-09-17T01:46:24.220 INFO:teuthology.task.ansible.out:skipping: [smithi045.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T01:46:24.288 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-09-17T01:46:24.289 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-09-17T01:46:24.290 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-09-17T01:46:24.292 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-17T01:46:24.307 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Friday 17 September 2021 01:46:24 +0000 (0:00:00.116) 0:08:16.315 ****** 2021-09-17T01:46:24.981 INFO:teuthology.task.ansible.out:ok: [smithi013.front.sepia.ceph.com] 2021-09-17T01:46:25.004 INFO:teuthology.task.ansible.out:ok: [smithi045.front.sepia.ceph.com] 2021-09-17T01:46:25.023 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Friday 17 September 2021 01:46:25 +0000 (0:00:00.716) 0:08:17.032 ****** 2021-09-17T01:46:25.229 INFO:teuthology.task.ansible.out:changed: [smithi013.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-17T01:46:25.251 INFO:teuthology.task.ansible.out:changed: [smithi045.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-17T01:46:25.255 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi013.front.sepia.ceph.com : ok=142 changed=42 unreachable=0 failed=0 skipped=178 rescued=0 ignored=4 smithi045.front.sepia.ceph.com : ok=142 changed=42 unreachable=0 failed=0 skipped=178 rescued=0 ignored=4 Friday 17 September 2021 01:46:25 +0000 (0:00:00.230) 0:08:17.263 ****** =============================================================================== Create all admin users with sudo access. ------------------------------- 61.50s common : nrpe - Ensure SELinux policy is up to date -------------------- 53.35s users : Update authorized_keys using the keys repo --------------------- 37.21s testnode : Install packages -------------------------------------------- 33.16s testnode : Ensure ceph packages are not present. ----------------------- 30.67s testnode : Ensure ceph dependency packages are not present. ------------ 28.86s container-host : Install container packages ---------------------------- 20.83s Remove revoked users --------------------------------------------------- 19.97s common : Register with subscription-manager. --------------------------- 18.24s common : nrpe - Remove SELinux policy package -------------------------- 15.16s common : Install Kerberos Packages (RedHat) ---------------------------- 15.11s common : nrpe - Load SELinux policy package ---------------------------- 14.87s testnode : Install epel packages --------------------------------------- 14.30s container-host : Ensure git is installed ------------------------------- 14.27s 2021-09-17T01:46:25.255 INFO:teuthology.task.ansible.out:testnode : Zap all non-root disks --------------------------------------- 7.68s common : Enable necessary rhsm repos ------------------------------------ 3.82s testnode : Remove packages ---------------------------------------------- 3.22s testnode : Make sure rpm dependencies are installed --------------------- 3.17s common : Get list of enabled RHSM repos ------ 2021-09-17T01:46:25.257 INFO:teuthology.task.ansible.out:--------------------------- 3.05s common : Make sure smartmontools is installed --------------------------- 2.93s 2021-09-17T01:46:25.660 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi013.front.sepia.ceph.com'), Remote(name='ubuntu@smithi045.front.sepia.ceph.com')] 2021-09-17T01:46:25.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi013.front.sepia.ceph.com' 2021-09-17T01:46:25.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi013.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T01:46:25.774 DEBUG:teuthology.orchestra.run.smithi013:> true 2021-09-17T01:46:25.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2021-09-17T01:46:25.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T01:46:25.915 DEBUG:teuthology.orchestra.run.smithi045:> true 2021-09-17T01:46:25.984 INFO:teuthology.run_tasks:Running task clock... 2021-09-17T01:46:26.003 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-09-17T01:46:26.004 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-17T01:46:26.007 DEBUG:teuthology.orchestra.run.smithi013:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-17T01:46:26.040 INFO:teuthology.orchestra.run.smithi013.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-09-17T01:46:26.057 INFO:teuthology.orchestra.run.smithi013.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-09-17T01:46:26.094 INFO:teuthology.orchestra.run.smithi013.stderr:sudo: ntpd: command not found 2021-09-17T01:46:26.105 INFO:teuthology.orchestra.run.smithi013.stdout:506 Cannot talk to daemon 2021-09-17T01:46:26.121 INFO:teuthology.orchestra.run.smithi013.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-09-17T01:46:26.136 INFO:teuthology.orchestra.run.smithi013.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-09-17T01:46:26.197 INFO:teuthology.orchestra.run.smithi013.stderr:bash: ntpq: command not found 2021-09-17T01:46:26.200 INFO:teuthology.orchestra.run.smithi013.stdout:210 Number of sources = 4 2021-09-17T01:46:26.200 INFO:teuthology.orchestra.run.smithi013.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-09-17T01:46:26.200 INFO:teuthology.orchestra.run.smithi013.stdout:=============================================================================== 2021-09-17T01:46:26.200 INFO:teuthology.orchestra.run.smithi013.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-17T01:46:26.201 INFO:teuthology.orchestra.run.smithi013.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-17T01:46:26.201 INFO:teuthology.orchestra.run.smithi013.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-17T01:46:26.201 INFO:teuthology.orchestra.run.smithi013.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-17T01:46:26.202 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-17T01:46:26.203 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-17T01:46:26.242 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-09-17T01:46:26.257 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-09-17T01:46:26.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2021-09-17T01:46:26.307 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2021-09-17T01:46:26.322 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-09-17T01:46:26.337 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-09-17T01:46:26.405 INFO:teuthology.orchestra.run.smithi045.stderr:bash: ntpq: command not found 2021-09-17T01:46:26.408 INFO:teuthology.orchestra.run.smithi045.stdout:210 Number of sources = 4 2021-09-17T01:46:26.408 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-09-17T01:46:26.408 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2021-09-17T01:46:26.408 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-17T01:46:26.409 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-17T01:46:26.409 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-17T01:46:26.409 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-17T01:46:26.409 INFO:teuthology.run_tasks:Running task install... 2021-09-17T01:46:26.419 DEBUG:teuthology.task.install:project ceph 2021-09-17T01:46:26.419 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049'}} 2021-09-17T01:46:26.420 DEBUG:teuthology.task.install:config {'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049'} 2021-09-17T01:46:26.420 INFO:teuthology.task.install:Using flavor: basic 2021-09-17T01:46:26.427 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-09-17T01:46:26.427 INFO:teuthology.task.install:extra packages: [] 2021-09-17T01:46:26.428 DEBUG:teuthology.orchestra.run.smithi013:> sudo lsb_release -is 2021-09-17T01:46:26.470 INFO:teuthology.orchestra.run.smithi013.stdout:RedHatEnterprise 2021-09-17T01:46:26.471 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-17T01:46:26.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo lsb_release -is 2021-09-17T01:46:26.473 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049', 'tag': None, 'wait_for_package': False} 2021-09-17T01:46:26.474 DEBUG:teuthology.orchestra.run.smithi013:> sudo lsb_release -is 2021-09-17T01:46:26.514 INFO:teuthology.orchestra.run.smithi045.stdout:RedHatEnterprise 2021-09-17T01:46:26.516 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-17T01:46:26.522 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049', 'tag': None, 'wait_for_package': False} 2021-09-17T01:46:26.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo lsb_release -is 2021-09-17T01:46:26.551 INFO:teuthology.orchestra.run.smithi013.stdout:RedHatEnterprise 2021-09-17T01:46:26.552 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-17T01:46:26.553 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2a911e49300877ce783c2becf9f484a85fa1c049 2021-09-17T01:46:26.598 INFO:teuthology.orchestra.run.smithi045.stdout:RedHatEnterprise 2021-09-17T01:46:26.599 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-17T01:46:26.599 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2a911e49300877ce783c2becf9f484a85fa1c049 2021-09-17T01:46:26.727 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-eric-testing-2/2a911e49300877ce783c2becf9f484a85fa1c049/centos/8/flavors/default/ 2021-09-17T01:46:26.727 INFO:teuthology.task.install.rpm:Package version is 17.0.0-7730.g2a911e49 2021-09-17T01:46:26.771 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-eric-testing-2/2a911e49300877ce783c2becf9f484a85fa1c049/centos/8/flavors/default/ 2021-09-17T01:46:26.771 INFO:teuthology.task.install.rpm:Package version is 17.0.0-7730.g2a911e49 2021-09-17T01:46:26.850 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-eric-testing-2/2a911e49300877ce783c2becf9f484a85fa1c049/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-eric-testing-2/2a911e49300877ce783c2becf9f484a85fa1c049/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-eric-testing-2/2a911e49300877ce783c2becf9f484a85fa1c049/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-09-17T01:46:26.850 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:46:26.851 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-09-17T01:46:26.878 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-09-17T01:46:26.878 DEBUG:teuthology.orchestra.run.smithi013:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2a911e49300877ce783c2becf9f484a85fa1c049/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-09-17T01:46:26.886 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-eric-testing-2/2a911e49300877ce783c2becf9f484a85fa1c049/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-eric-testing-2/2a911e49300877ce783c2becf9f484a85fa1c049/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-eric-testing-2/2a911e49300877ce783c2becf9f484a85fa1c049/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-09-17T01:46:26.887 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:46:26.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-09-17T01:46:26.913 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-09-17T01:46:26.914 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2a911e49300877ce783c2becf9f484a85fa1c049/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-09-17T01:46:26.944 DEBUG:teuthology.orchestra.run.smithi013:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-09-17T01:46:26.980 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-09-17T01:46:27.020 DEBUG:teuthology.orchestra.run.smithi013:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-09-17T01:46:27.057 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-09-17T01:46:27.084 INFO:teuthology.orchestra.run.smithi013.stdout:check_obsoletes = 1 2021-09-17T01:46:27.086 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum clean all 2021-09-17T01:46:27.122 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2021-09-17T01:46:27.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2021-09-17T01:46:27.461 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:46:27.500 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:46:27.800 INFO:teuthology.orchestra.run.smithi013.stdout:62 files removed 2021-09-17T01:46:27.828 INFO:teuthology.orchestra.run.smithi045.stdout:62 files removed 2021-09-17T01:46:27.851 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install ceph-radosgw 2021-09-17T01:46:27.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw 2021-09-17T01:46:28.187 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:46:28.208 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:46:28.884 INFO:teuthology.orchestra.run.smithi013.stdout:ceph packages for x86_64 296 kB/s | 75 kB 00:00 2021-09-17T01:46:28.889 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 310 kB/s | 75 kB 00:00 2021-09-17T01:46:29.116 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 46 kB/s | 9.0 kB 00:00 2021-09-17T01:46:29.132 INFO:teuthology.orchestra.run.smithi013.stdout:ceph noarch packages 42 kB/s | 9.0 kB 00:00 2021-09-17T01:46:29.329 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 7.4 kB/s | 1.4 kB 00:00 2021-09-17T01:46:29.364 INFO:teuthology.orchestra.run.smithi013.stdout:ceph source packages 6.7 kB/s | 1.4 kB 00:00 2021-09-17T01:46:31.527 INFO:teuthology.orchestra.run.smithi013.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2021-09-17T01:46:31.540 INFO:teuthology.orchestra.run.smithi045.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2021-09-17T01:46:32.373 INFO:teuthology.orchestra.run.smithi013.stdout:Extra Packages for Enterprise Linux 13 MB/s | 10 MB 00:00 2021-09-17T01:46:32.567 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 10 MB/s | 10 MB 00:01 2021-09-17T01:46:34.613 INFO:teuthology.orchestra.run.smithi013.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2021-09-17T01:46:34.696 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2021-09-17T01:46:34.987 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 102 MB/s | 32 MB 00:00 2021-09-17T01:46:35.071 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 102 MB/s | 32 MB 00:00 2021-09-17T01:46:40.408 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 MB/s | 37 MB 00:00 2021-09-17T01:46:40.582 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 114 MB/s | 37 MB 00:00 2021-09-17T01:46:49.225 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:46:49.228 INFO:teuthology.orchestra.run.smithi045.stdout:==================================================================================================== 2021-09-17T01:46:49.228 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-09-17T01:46:49.228 INFO:teuthology.orchestra.run.smithi045.stdout:==================================================================================================== 2021-09-17T01:46:49.229 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:46:49.229 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 8.4 M 2021-09-17T01:46:49.229 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2021-09-17T01:46:49.230 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 3.4 M 2021-09-17T01:46:49.230 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 3.9 M 2021-09-17T01:46:49.230 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-09-17T01:46:49.230 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 5.2 M 2021-09-17T01:46:49.231 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 20 M 2021-09-17T01:46:49.231 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 24 k 2021-09-17T01:46:49.232 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-09-17T01:46:49.232 INFO:teuthology.orchestra.run.smithi045.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-09-17T01:46:49.232 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 782 k 2021-09-17T01:46:49.233 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-09-17T01:46:49.233 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-09-17T01:46:49.233 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 482 k 2021-09-17T01:46:49.233 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-09-17T01:46:49.233 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 3.7 M 2021-09-17T01:46:49.234 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-09-17T01:46:49.234 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-09-17T01:46:49.234 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 45 k 2021-09-17T01:46:49.234 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 78 k 2021-09-17T01:46:49.235 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 177 k 2021-09-17T01:46:49.235 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-09-17T01:46:49.235 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 365 k 2021-09-17T01:46:49.236 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 345 k 2021-09-17T01:46:49.236 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 109 k 2021-09-17T01:46:49.236 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:46:49.236 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:46:49.237 INFO:teuthology.orchestra.run.smithi045.stdout:==================================================================================================== 2021-09-17T01:46:49.237 INFO:teuthology.orchestra.run.smithi045.stdout:Install 21 Packages 2021-09-17T01:46:49.237 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 2 Packages 2021-09-17T01:46:49.237 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:46:49.238 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 48 M 2021-09-17T01:46:49.238 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:46:49.422 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:46:49.425 INFO:teuthology.orchestra.run.smithi013.stdout:==================================================================================================== 2021-09-17T01:46:49.425 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repository Size 2021-09-17T01:46:49.425 INFO:teuthology.orchestra.run.smithi013.stdout:==================================================================================================== 2021-09-17T01:46:49.426 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:46:49.426 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-radosgw x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 8.4 M 2021-09-17T01:46:49.426 INFO:teuthology.orchestra.run.smithi013.stdout:Upgrading: 2021-09-17T01:46:49.426 INFO:teuthology.orchestra.run.smithi013.stdout: librados2 x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 3.4 M 2021-09-17T01:46:49.427 INFO:teuthology.orchestra.run.smithi013.stdout: librbd1 x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 3.9 M 2021-09-17T01:46:49.427 INFO:teuthology.orchestra.run.smithi013.stdout:Installing dependencies: 2021-09-17T01:46:49.427 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-base x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 5.2 M 2021-09-17T01:46:49.427 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-common x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 20 M 2021-09-17T01:46:49.427 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-selinux x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 24 k 2021-09-17T01:46:49.428 INFO:teuthology.orchestra.run.smithi013.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-09-17T01:46:49.428 INFO:teuthology.orchestra.run.smithi013.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-09-17T01:46:49.428 INFO:teuthology.orchestra.run.smithi013.stdout: libcephfs2 x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 782 k 2021-09-17T01:46:49.428 INFO:teuthology.orchestra.run.smithi013.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-09-17T01:46:49.429 INFO:teuthology.orchestra.run.smithi013.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-09-17T01:46:49.429 INFO:teuthology.orchestra.run.smithi013.stdout: libradosstriper1 x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 482 k 2021-09-17T01:46:49.429 INFO:teuthology.orchestra.run.smithi013.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-09-17T01:46:49.429 INFO:teuthology.orchestra.run.smithi013.stdout: librgw2 x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 3.7 M 2021-09-17T01:46:49.429 INFO:teuthology.orchestra.run.smithi013.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-09-17T01:46:49.430 INFO:teuthology.orchestra.run.smithi013.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-09-17T01:46:49.430 INFO:teuthology.orchestra.run.smithi013.stdout: python3-ceph-argparse x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 45 k 2021-09-17T01:46:49.430 INFO:teuthology.orchestra.run.smithi013.stdout: python3-ceph-common x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 78 k 2021-09-17T01:46:49.430 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cephfs x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 177 k 2021-09-17T01:46:49.430 INFO:teuthology.orchestra.run.smithi013.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-09-17T01:46:49.431 INFO:teuthology.orchestra.run.smithi013.stdout: python3-rados x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 365 k 2021-09-17T01:46:49.431 INFO:teuthology.orchestra.run.smithi013.stdout: python3-rbd x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 345 k 2021-09-17T01:46:49.431 INFO:teuthology.orchestra.run.smithi013.stdout: python3-rgw x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 109 k 2021-09-17T01:46:49.431 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:46:49.432 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:46:49.432 INFO:teuthology.orchestra.run.smithi013.stdout:==================================================================================================== 2021-09-17T01:46:49.432 INFO:teuthology.orchestra.run.smithi013.stdout:Install 21 Packages 2021-09-17T01:46:49.433 INFO:teuthology.orchestra.run.smithi013.stdout:Upgrade 2 Packages 2021-09-17T01:46:49.433 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:46:49.433 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 48 M 2021-09-17T01:46:49.434 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:46:50.655 INFO:teuthology.orchestra.run.smithi013.stdout:(1/23): ceph-base-17.0.0-7730.g2a911e49.el8.x86 4.3 MB/s | 5.2 MB 00:01 2021-09-17T01:46:50.678 INFO:teuthology.orchestra.run.smithi013.stdout:(2/23): ceph-selinux-17.0.0-7730.g2a911e49.el8. 1.0 MB/s | 24 kB 00:00 2021-09-17T01:46:50.742 INFO:teuthology.orchestra.run.smithi013.stdout:(3/23): libcephfs2-17.0.0-7730.g2a911e49.el8.x8 12 MB/s | 782 kB 00:00 2021-09-17T01:46:50.785 INFO:teuthology.orchestra.run.smithi013.stdout:(4/23): libradosstriper1-17.0.0-7730.g2a911e49. 11 MB/s | 482 kB 00:00 2021-09-17T01:46:51.027 INFO:teuthology.orchestra.run.smithi013.stdout:(5/23): librgw2-17.0.0-7730.g2a911e49.el8.x86_6 15 MB/s | 3.7 MB 00:00 2021-09-17T01:46:51.048 INFO:teuthology.orchestra.run.smithi013.stdout:(6/23): python3-ceph-argparse-17.0.0-7730.g2a91 2.1 MB/s | 45 kB 00:00 2021-09-17T01:46:51.070 INFO:teuthology.orchestra.run.smithi013.stdout:(7/23): python3-ceph-common-17.0.0-7730.g2a911e 3.6 MB/s | 78 kB 00:00 2021-09-17T01:46:51.092 INFO:teuthology.orchestra.run.smithi013.stdout:(8/23): python3-cephfs-17.0.0-7730.g2a911e49.el 7.9 MB/s | 177 kB 00:00 2021-09-17T01:46:51.123 INFO:teuthology.orchestra.run.smithi045.stdout:(1/23): ceph-base-17.0.0-7730.g2a911e49.el8.x86 2.8 MB/s | 5.2 MB 00:01 2021-09-17T01:46:51.124 INFO:teuthology.orchestra.run.smithi013.stdout:(9/23): python3-rados-17.0.0-7730.g2a911e49.el8 11 MB/s | 365 kB 00:00 2021-09-17T01:46:51.153 INFO:teuthology.orchestra.run.smithi045.stdout:(2/23): ceph-selinux-17.0.0-7730.g2a911e49.el8. 835 kB/s | 24 kB 00:00 2021-09-17T01:46:51.161 INFO:teuthology.orchestra.run.smithi013.stdout:(10/23): python3-rbd-17.0.0-7730.g2a911e49.el8. 9.2 MB/s | 345 kB 00:00 2021-09-17T01:46:51.189 INFO:teuthology.orchestra.run.smithi013.stdout:(11/23): python3-rgw-17.0.0-7730.g2a911e49.el8. 4.1 MB/s | 109 kB 00:00 2021-09-17T01:46:51.325 INFO:teuthology.orchestra.run.smithi045.stdout:(3/23): libcephfs2-17.0.0-7730.g2a911e49.el8.x8 4.4 MB/s | 782 kB 00:00 2021-09-17T01:46:51.435 INFO:teuthology.orchestra.run.smithi045.stdout:(4/23): libradosstriper1-17.0.0-7730.g2a911e49. 4.3 MB/s | 482 kB 00:00 2021-09-17T01:46:51.810 INFO:teuthology.orchestra.run.smithi013.stdout:(12/23): gperftools-libs-2.7-9.el8.x86_64.rpm 492 kB/s | 306 kB 00:00 2021-09-17T01:46:51.929 INFO:teuthology.orchestra.run.smithi013.stdout:(13/23): leveldb-1.22-1.el8.x86_64.rpm 1.5 MB/s | 181 kB 00:00 2021-09-17T01:46:51.970 INFO:teuthology.orchestra.run.smithi013.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2021-09-17T01:46:52.053 INFO:teuthology.orchestra.run.smithi013.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 907 kB/s | 75 kB 00:00 2021-09-17T01:46:52.093 INFO:teuthology.orchestra.run.smithi013.stdout:(16/23): librdkafka-0.11.4-1.el8.x86_64.rpm 8.6 MB/s | 353 kB 00:00 2021-09-17T01:46:52.121 INFO:teuthology.orchestra.run.smithi013.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 9.3 MB/s | 259 kB 00:00 2021-09-17T01:46:52.145 INFO:teuthology.orchestra.run.smithi013.stdout:(18/23): python3-prettytable-0.7.2-14.el8.noarc 1.7 MB/s | 44 kB 00:00 2021-09-17T01:46:52.172 INFO:teuthology.orchestra.run.smithi013.stdout:(19/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.7 MB/s | 47 kB 00:00 2021-09-17T01:46:52.467 INFO:teuthology.orchestra.run.smithi013.stdout:(20/23): ceph-radosgw-17.0.0-7730.g2a911e49.el8 2.8 MB/s | 8.4 MB 00:03 2021-09-17T01:46:52.946 INFO:teuthology.orchestra.run.smithi013.stdout:(21/23): librados2-17.0.0-7730.g2a911e49.el8.x8 4.4 MB/s | 3.4 MB 00:00 2021-09-17T01:46:53.016 INFO:teuthology.orchestra.run.smithi045.stdout:(5/23): ceph-common-17.0.0-7730.g2a911e49.el8.x 5.2 MB/s | 20 MB 00:03 2021-09-17T01:46:53.030 INFO:teuthology.orchestra.run.smithi045.stdout:(6/23): librgw2-17.0.0-7730.g2a911e49.el8.x86_6 2.3 MB/s | 3.7 MB 00:01 2021-09-17T01:46:53.052 INFO:teuthology.orchestra.run.smithi045.stdout:(7/23): python3-ceph-argparse-17.0.0-7730.g2a91 1.2 MB/s | 45 kB 00:00 2021-09-17T01:46:53.065 INFO:teuthology.orchestra.run.smithi045.stdout:(8/23): python3-ceph-common-17.0.0-7730.g2a911e 2.2 MB/s | 78 kB 00:00 2021-09-17T01:46:53.104 INFO:teuthology.orchestra.run.smithi045.stdout:(9/23): python3-cephfs-17.0.0-7730.g2a911e49.el 3.4 MB/s | 177 kB 00:00 2021-09-17T01:46:53.167 INFO:teuthology.orchestra.run.smithi045.stdout:(10/23): python3-rbd-17.0.0-7730.g2a911e49.el8. 5.4 MB/s | 345 kB 00:00 2021-09-17T01:46:53.185 INFO:teuthology.orchestra.run.smithi045.stdout:(11/23): python3-rados-17.0.0-7730.g2a911e49.el 3.0 MB/s | 365 kB 00:00 2021-09-17T01:46:53.202 INFO:teuthology.orchestra.run.smithi045.stdout:(12/23): python3-rgw-17.0.0-7730.g2a911e49.el8. 3.1 MB/s | 109 kB 00:00 2021-09-17T01:46:53.319 INFO:teuthology.orchestra.run.smithi013.stdout:(22/23): ceph-common-17.0.0-7730.g2a911e49.el8. 5.0 MB/s | 20 MB 00:03 2021-09-17T01:46:53.376 INFO:teuthology.orchestra.run.smithi045.stdout:(13/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2021-09-17T01:46:53.377 INFO:teuthology.orchestra.run.smithi045.stdout:(14/23): leveldb-1.22-1.el8.x86_64.rpm 1.0 MB/s | 181 kB 00:00 2021-09-17T01:46:53.409 INFO:teuthology.orchestra.run.smithi045.stdout:(15/23): liboath-2.6.2-3.el8.x86_64.rpm 1.8 MB/s | 59 kB 00:00 2021-09-17T01:46:53.422 INFO:teuthology.orchestra.run.smithi045.stdout:(16/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.6 MB/s | 75 kB 00:00 2021-09-17T01:46:53.459 INFO:teuthology.orchestra.run.smithi045.stdout:(17/23): librdkafka-0.11.4-1.el8.x86_64.rpm 6.9 MB/s | 353 kB 00:00 2021-09-17T01:46:53.479 INFO:teuthology.orchestra.run.smithi045.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.6 MB/s | 259 kB 00:00 2021-09-17T01:46:53.495 INFO:teuthology.orchestra.run.smithi045.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 1.5 MB/s | 44 kB 00:00 2021-09-17T01:46:53.511 INFO:teuthology.orchestra.run.smithi045.stdout:(20/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2021-09-17T01:46:53.672 INFO:teuthology.orchestra.run.smithi013.stdout:(23/23): librbd1-17.0.0-7730.g2a911e49.el8.x86_ 3.2 MB/s | 3.9 MB 00:01 2021-09-17T01:46:53.676 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:46:53.676 INFO:teuthology.orchestra.run.smithi013.stdout:Total 11 MB/s | 48 MB 00:04 2021-09-17T01:46:53.681 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:46:53.734 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:46:53.734 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:46:53.886 INFO:teuthology.orchestra.run.smithi045.stdout:(21/23): librbd1-17.0.0-7730.g2a911e49.el8.x86_ 10 MB/s | 3.9 MB 00:00 2021-09-17T01:46:54.019 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:46:54.080 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:46:54.464 INFO:teuthology.orchestra.run.smithi045.stdout:(22/23): ceph-radosgw-17.0.0-7730.g2a911e49.el8 1.6 MB/s | 8.4 MB 00:05 2021-09-17T01:46:54.644 INFO:teuthology.orchestra.run.smithi045.stdout:(23/23): librados2-17.0.0-7730.g2a911e49.el8.x8 3.0 MB/s | 3.4 MB 00:01 2021-09-17T01:46:54.647 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:46:54.647 INFO:teuthology.orchestra.run.smithi045.stdout:Total 8.8 MB/s | 48 MB 00:05 2021-09-17T01:46:54.653 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:46:54.703 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:46:54.704 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:46:54.991 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:46:54.995 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:46:55.058 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:46:55.202 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-17T01:46:55.620 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-17T01:46:55.797 INFO:teuthology.orchestra.run.smithi013.stdout: Upgrading : librados2-2:17.0.0-7730.g2a911e49.el8.x86_64 2/25 2021-09-17T01:46:55.880 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: librados2-2:17.0.0-7730.g2a911e49.el8.x86_64 2/25 2021-09-17T01:46:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:46:56.041 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-rados-2:17.0.0-7730.g2a911e49.el8.x86_64 3/25 2021-09-17T01:46:56.166 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-17T01:46:56.263 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2021-09-17T01:46:56.544 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : libcephfs2-2:17.0.0-7730.g2a911e49.el8.x86_64 5/25 2021-09-17T01:46:56.569 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-17T01:46:56.759 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:17.0.0-7730.g2a911e49.el8.x86_64 2/25 2021-09-17T01:46:56.855 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:17.0.0-7730.g2a911e49.el8.x86_64 2/25 2021-09-17T01:46:56.977 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: libcephfs2-2:17.0.0-7730.g2a911e49.el8.x86_64 5/25 2021-09-17T01:46:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:17.0.0-7730.g2a911e49.el8.x86_64 3/25 2021-09-17T01:46:57.161 INFO:teuthology.orchestra.run.smithi013.stdout: Upgrading : librbd1-2:17.0.0-7730.g2a911e49.el8.x86_64 6/25 2021-09-17T01:46:57.227 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: librbd1-2:17.0.0-7730.g2a911e49.el8.x86_64 6/25 2021-09-17T01:46:57.269 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2021-09-17T01:46:57.395 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2021-09-17T01:46:57.466 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:17.0.0-7730.g2a911e49.el8.x86_64 5/25 2021-09-17T01:46:57.511 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-17T01:46:57.922 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:17.0.0-7730.g2a911e49.el8.x86_64 5/25 2021-09-17T01:46:58.043 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-17T01:46:58.116 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:17.0.0-7730.g2a911e49.el8.x86_64 6/25 2021-09-17T01:46:58.168 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : librgw2-2:17.0.0-7730.g2a911e49.el8.x86_64 9/25 2021-09-17T01:46:58.184 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:17.0.0-7730.g2a911e49.el8.x86_64 6/25 2021-09-17T01:46:58.251 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: librgw2-2:17.0.0-7730.g2a911e49.el8.x86_64 9/25 2021-09-17T01:46:58.386 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2021-09-17T01:46:58.402 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-09-17T01:46:58.521 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-ceph-argparse-2:17.0.0-7730.g2a911e49.el8. 11/25 2021-09-17T01:46:58.658 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-17T01:46:58.672 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-cephfs-2:17.0.0-7730.g2a911e49.el8.x86_64 12/25 2021-09-17T01:46:58.905 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-rgw-2:17.0.0-7730.g2a911e49.el8.x86_64 13/25 2021-09-17T01:46:59.090 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-rbd-2:17.0.0-7730.g2a911e49.el8.x86_64 14/25 2021-09-17T01:46:59.098 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-17T01:46:59.210 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : libradosstriper1-2:17.0.0-7730.g2a911e49.el8.x86_6 15/25 2021-09-17T01:46:59.233 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:17.0.0-7730.g2a911e49.el8.x86_64 9/25 2021-09-17T01:46:59.274 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: libradosstriper1-2:17.0.0-7730.g2a911e49.el8.x86_6 15/25 2021-09-17T01:46:59.318 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:17.0.0-7730.g2a911e49.el8.x86_64 9/25 2021-09-17T01:46:59.387 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-09-17T01:46:59.489 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-09-17T01:46:59.564 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-17T01:46:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:17.0.0-7730.g2a911e49.el8. 11/25 2021-09-17T01:46:59.748 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-09-17T01:46:59.820 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:17.0.0-7730.g2a911e49.el8.x86_64 12/25 2021-09-17T01:46:59.928 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-ceph-common-2:17.0.0-7730.g2a911e49.el8.x8 19/25 2021-09-17T01:47:00.070 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:17.0.0-7730.g2a911e49.el8.x86_64 13/25 2021-09-17T01:47:00.280 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:17.0.0-7730.g2a911e49.el8.x86_64 14/25 2021-09-17T01:47:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:17.0.0-7730.g2a911e49.el8.x86_6 15/25 2021-09-17T01:47:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:17.0.0-7730.g2a911e49.el8.x86_6 15/25 2021-09-17T01:47:00.652 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-09-17T01:47:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-17T01:47:01.107 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-09-17T01:47:01.283 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:17.0.0-7730.g2a911e49.el8.x8 19/25 2021-09-17T01:47:04.320 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 20/25 2021-09-17T01:47:04.485 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 20/25 2021-09-17T01:47:05.088 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 20/25 2021-09-17T01:47:05.313 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-base-2:17.0.0-7730.g2a911e49.el8.x86_64 21/25 2021-09-17T01:47:05.427 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-base-2:17.0.0-7730.g2a911e49.el8.x86_64 21/25 2021-09-17T01:47:05.577 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-selinux-2:17.0.0-7730.g2a911e49.el8.x86_64 22/25 2021-09-17T01:47:06.097 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 20/25 2021-09-17T01:47:06.340 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 20/25 2021-09-17T01:47:06.946 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 20/25 2021-09-17T01:47:07.316 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:17.0.0-7730.g2a911e49.el8.x86_64 21/25 2021-09-17T01:47:07.453 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:17.0.0-7730.g2a911e49.el8.x86_64 21/25 2021-09-17T01:47:07.565 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:17.0.0-7730.g2a911e49.el8.x86_64 22/25 2021-09-17T01:47:23.722 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-selinux-2:17.0.0-7730.g2a911e49.el8.x86_64 22/25 2021-09-17T01:47:23.723 INFO:teuthology.orchestra.run.smithi013.stdout:skipping the directory /sys 2021-09-17T01:47:23.723 INFO:teuthology.orchestra.run.smithi013.stdout:skipping the directory /proc 2021-09-17T01:47:23.724 INFO:teuthology.orchestra.run.smithi013.stdout:skipping the directory /dev 2021-09-17T01:47:23.724 INFO:teuthology.orchestra.run.smithi013.stdout:skipping the directory /run 2021-09-17T01:47:23.724 INFO:teuthology.orchestra.run.smithi013.stdout:skipping the directory /mnt 2021-09-17T01:47:23.724 INFO:teuthology.orchestra.run.smithi013.stdout:skipping the directory /var/tmp 2021-09-17T01:47:23.725 INFO:teuthology.orchestra.run.smithi013.stdout:skipping the directory /home 2021-09-17T01:47:23.725 INFO:teuthology.orchestra.run.smithi013.stdout:skipping the directory /tmp 2021-09-17T01:47:23.725 INFO:teuthology.orchestra.run.smithi013.stdout:skipping the directory /dev 2021-09-17T01:47:23.725 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:47:24.836 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-radosgw-2:17.0.0-7730.g2a911e49.el8.x86_64 23/25 2021-09-17T01:47:24.936 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-7730.g2a911e49.el8.x86_64 23/25 2021-09-17T01:47:24.936 INFO:teuthology.orchestra.run.smithi013.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-17T01:47:25.251 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-17T01:47:25.252 INFO:teuthology.orchestra.run.smithi013.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-17T01:47:26.016 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-17T01:47:26.016 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-base-2:17.0.0-7730.g2a911e49.el8.x86_64 1/25 2021-09-17T01:47:26.017 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 2/25 2021-09-17T01:47:26.017 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-radosgw-2:17.0.0-7730.g2a911e49.el8.x86_64 3/25 2021-09-17T01:47:26.017 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-selinux-2:17.0.0-7730.g2a911e49.el8.x86_64 4/25 2021-09-17T01:47:26.017 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : libcephfs2-2:17.0.0-7730.g2a911e49.el8.x86_64 5/25 2021-09-17T01:47:26.018 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : libradosstriper1-2:17.0.0-7730.g2a911e49.el8.x86_6 6/25 2021-09-17T01:47:26.018 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : librgw2-2:17.0.0-7730.g2a911e49.el8.x86_64 7/25 2021-09-17T01:47:26.018 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-ceph-argparse-2:17.0.0-7730.g2a911e49.el8. 8/25 2021-09-17T01:47:26.018 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-ceph-common-2:17.0.0-7730.g2a911e49.el8.x8 9/25 2021-09-17T01:47:26.019 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-cephfs-2:17.0.0-7730.g2a911e49.el8.x86_64 10/25 2021-09-17T01:47:26.019 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-rados-2:17.0.0-7730.g2a911e49.el8.x86_64 11/25 2021-09-17T01:47:26.019 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-rbd-2:17.0.0-7730.g2a911e49.el8.x86_64 12/25 2021-09-17T01:47:26.019 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-rgw-2:17.0.0-7730.g2a911e49.el8.x86_64 13/25 2021-09-17T01:47:26.019 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2021-09-17T01:47:26.020 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2021-09-17T01:47:26.020 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2021-09-17T01:47:26.020 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-17T01:47:26.020 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 18/25 2021-09-17T01:47:26.020 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2021-09-17T01:47:26.021 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2021-09-17T01:47:26.021 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2021-09-17T01:47:26.021 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : librados2-2:17.0.0-7730.g2a911e49.el8.x86_64 22/25 2021-09-17T01:47:26.021 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-09-17T01:47:26.022 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : librbd1-2:17.0.0-7730.g2a911e49.el8.x86_64 24/25 2021-09-17T01:47:26.029 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:17.0.0-7730.g2a911e49.el8.x86_64 22/25 2021-09-17T01:47:26.029 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2021-09-17T01:47:26.030 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2021-09-17T01:47:26.030 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /dev 2021-09-17T01:47:26.030 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /run 2021-09-17T01:47:26.031 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2021-09-17T01:47:26.036 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2021-09-17T01:47:26.037 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2021-09-17T01:47:26.037 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2021-09-17T01:47:26.037 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /dev 2021-09-17T01:47:26.037 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:47:26.862 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-09-17T01:47:26.863 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:47:27.164 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:17.0.0-7730.g2a911e49.el8.x86_64 23/25 2021-09-17T01:47:27.240 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-7730.g2a911e49.el8.x86_64 23/25 2021-09-17T01:47:27.240 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-17T01:47:27.456 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-17T01:47:27.456 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-17T01:47:28.193 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-17T01:47:28.194 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:17.0.0-7730.g2a911e49.el8.x86_64 1/25 2021-09-17T01:47:28.194 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 2/25 2021-09-17T01:47:28.194 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:17.0.0-7730.g2a911e49.el8.x86_64 3/25 2021-09-17T01:47:28.195 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:17.0.0-7730.g2a911e49.el8.x86_64 4/25 2021-09-17T01:47:28.195 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:17.0.0-7730.g2a911e49.el8.x86_64 5/25 2021-09-17T01:47:28.195 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:17.0.0-7730.g2a911e49.el8.x86_6 6/25 2021-09-17T01:47:28.195 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:17.0.0-7730.g2a911e49.el8.x86_64 7/25 2021-09-17T01:47:28.196 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:17.0.0-7730.g2a911e49.el8. 8/25 2021-09-17T01:47:28.196 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:17.0.0-7730.g2a911e49.el8.x8 9/25 2021-09-17T01:47:28.196 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:17.0.0-7730.g2a911e49.el8.x86_64 10/25 2021-09-17T01:47:28.196 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:17.0.0-7730.g2a911e49.el8.x86_64 11/25 2021-09-17T01:47:28.196 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:17.0.0-7730.g2a911e49.el8.x86_64 12/25 2021-09-17T01:47:28.197 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:17.0.0-7730.g2a911e49.el8.x86_64 13/25 2021-09-17T01:47:28.197 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2021-09-17T01:47:28.197 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2021-09-17T01:47:28.197 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2021-09-17T01:47:28.198 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-17T01:47:28.198 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 18/25 2021-09-17T01:47:28.198 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2021-09-17T01:47:28.198 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2021-09-17T01:47:28.199 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2021-09-17T01:47:28.199 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:17.0.0-7730.g2a911e49.el8.x86_64 22/25 2021-09-17T01:47:28.199 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-09-17T01:47:28.199 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:17.0.0-7730.g2a911e49.el8.x86_64 24/25 2021-09-17T01:47:29.035 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-09-17T01:47:29.036 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:47:38.691 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:47:32 AM UTC. 2021-09-17T01:47:40.636 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:47:35 AM UTC. 2021-09-17T01:47:41.410 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:47:41.410 INFO:teuthology.orchestra.run.smithi013.stdout:Upgraded: 2021-09-17T01:47:41.411 INFO:teuthology.orchestra.run.smithi013.stdout: librados2-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.411 INFO:teuthology.orchestra.run.smithi013.stdout: librbd1-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.411 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:47:41.411 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-base-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.411 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.412 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-radosgw-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.412 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-selinux-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.412 INFO:teuthology.orchestra.run.smithi013.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-09-17T01:47:41.412 INFO:teuthology.orchestra.run.smithi013.stdout: leveldb-1.22-1.el8.x86_64 2021-09-17T01:47:41.413 INFO:teuthology.orchestra.run.smithi013.stdout: libcephfs2-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.413 INFO:teuthology.orchestra.run.smithi013.stdout: liboath-2.6.2-3.el8.x86_64 2021-09-17T01:47:41.413 INFO:teuthology.orchestra.run.smithi013.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-09-17T01:47:41.413 INFO:teuthology.orchestra.run.smithi013.stdout: libradosstriper1-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.414 INFO:teuthology.orchestra.run.smithi013.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-09-17T01:47:41.414 INFO:teuthology.orchestra.run.smithi013.stdout: librgw2-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.414 INFO:teuthology.orchestra.run.smithi013.stdout: libunwind-1.3.1-3.el8.x86_64 2021-09-17T01:47:41.414 INFO:teuthology.orchestra.run.smithi013.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-09-17T01:47:41.415 INFO:teuthology.orchestra.run.smithi013.stdout: python3-ceph-argparse-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.415 INFO:teuthology.orchestra.run.smithi013.stdout: python3-ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.416 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cephfs-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.416 INFO:teuthology.orchestra.run.smithi013.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-09-17T01:47:41.416 INFO:teuthology.orchestra.run.smithi013.stdout: python3-rados-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.416 INFO:teuthology.orchestra.run.smithi013.stdout: python3-rbd-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.417 INFO:teuthology.orchestra.run.smithi013.stdout: python3-rgw-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:41.417 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:47:41.417 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:47:41.552 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install ceph-test 2021-09-17T01:47:41.902 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:47:42.439 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:47:43.220 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:47:43.221 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2021-09-17T01:47:43.221 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.221 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.222 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:47:43.222 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.222 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.222 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.223 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.223 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-09-17T01:47:43.223 INFO:teuthology.orchestra.run.smithi045.stdout: leveldb-1.22-1.el8.x86_64 2021-09-17T01:47:43.223 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.223 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.2-3.el8.x86_64 2021-09-17T01:47:43.224 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-09-17T01:47:43.224 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.224 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-09-17T01:47:43.224 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.224 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.3.1-3.el8.x86_64 2021-09-17T01:47:43.225 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-09-17T01:47:43.225 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.225 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.225 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.225 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-09-17T01:47:43.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.226 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:47:43.226 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:47:43.227 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:47:43.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-test 2021-09-17T01:47:43.710 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:47:44.230 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 71 kB/s | 2.8 kB 00:00 2021-09-17T01:47:47.489 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-17T01:47:49.925 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-17T01:47:53.088 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:47:55.722 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:47:55.723 INFO:teuthology.orchestra.run.smithi013.stdout:============================================================================================ 2021-09-17T01:47:55.724 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repository Size 2021-09-17T01:47:55.724 INFO:teuthology.orchestra.run.smithi013.stdout:============================================================================================ 2021-09-17T01:47:55.725 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:47:55.725 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-test x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 43 M 2021-09-17T01:47:55.726 INFO:teuthology.orchestra.run.smithi013.stdout:Installing dependencies: 2021-09-17T01:47:55.726 INFO:teuthology.orchestra.run.smithi013.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-09-17T01:47:55.727 INFO:teuthology.orchestra.run.smithi013.stdout: libcephsqlite x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 170 k 2021-09-17T01:47:55.727 INFO:teuthology.orchestra.run.smithi013.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-09-17T01:47:55.728 INFO:teuthology.orchestra.run.smithi013.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-09-17T01:47:55.728 INFO:teuthology.orchestra.run.smithi013.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-09-17T01:47:55.728 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:47:55.729 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:47:55.729 INFO:teuthology.orchestra.run.smithi013.stdout:============================================================================================ 2021-09-17T01:47:55.730 INFO:teuthology.orchestra.run.smithi013.stdout:Install 6 Packages 2021-09-17T01:47:55.730 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:47:55.731 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 44 M 2021-09-17T01:47:55.731 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 206 M 2021-09-17T01:47:55.731 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:47:55.841 INFO:teuthology.orchestra.run.smithi013.stdout:(1/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 596 kB/s | 69 kB 00:00 2021-09-17T01:47:55.876 INFO:teuthology.orchestra.run.smithi013.stdout:(2/6): libcephsqlite-17.0.0-7730.g2a911e49.el8. 1.1 MB/s | 170 kB 00:00 2021-09-17T01:47:55.887 INFO:teuthology.orchestra.run.smithi013.stdout:(3/6): jq-1.5-12.el8.x86_64.rpm 3.4 MB/s | 161 kB 00:00 2021-09-17T01:47:55.922 INFO:teuthology.orchestra.run.smithi013.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.0 MB/s | 187 kB 00:00 2021-09-17T01:47:55.930 INFO:teuthology.orchestra.run.smithi013.stdout:(5/6): socat-1.7.3.3-2.el8.x86_64.rpm 7.0 MB/s | 302 kB 00:00 2021-09-17T01:47:55.985 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:47:58.732 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:47:58.733 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2021-09-17T01:47:58.734 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-09-17T01:47:58.734 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2021-09-17T01:47:58.734 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:47:58.734 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 43 M 2021-09-17T01:47:58.735 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-09-17T01:47:58.735 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-09-17T01:47:58.735 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 170 k 2021-09-17T01:47:58.735 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-09-17T01:47:58.736 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-09-17T01:47:58.736 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-09-17T01:47:58.736 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:47:58.736 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:47:58.736 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2021-09-17T01:47:58.737 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2021-09-17T01:47:58.737 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:47:58.738 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 44 M 2021-09-17T01:47:58.738 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 206 M 2021-09-17T01:47:58.738 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:47:58.831 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 727 kB/s | 69 kB 00:00 2021-09-17T01:47:58.876 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 3.6 MB/s | 161 kB 00:00 2021-09-17T01:47:58.903 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 6.8 MB/s | 187 kB 00:00 2021-09-17T01:47:58.931 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 10 MB/s | 302 kB 00:00 2021-09-17T01:47:59.254 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): libcephsqlite-17.0.0-7730.g2a911e49.el8. 329 kB/s | 170 kB 00:00 2021-09-17T01:48:00.407 INFO:teuthology.orchestra.run.smithi013.stdout:(6/6): ceph-test-17.0.0-7730.g2a911e49.el8.x86_ 9.2 MB/s | 43 MB 00:04 2021-09-17T01:48:00.410 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:48:00.410 INFO:teuthology.orchestra.run.smithi013.stdout:Total 9.4 MB/s | 44 MB 00:04 2021-09-17T01:48:00.415 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:48:00.441 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:48:00.442 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:48:00.695 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:48:00.699 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:48:01.184 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:48:01.365 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2021-09-17T01:48:01.569 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-17T01:48:01.654 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-17T01:48:01.813 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2021-09-17T01:48:01.951 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 4/6 2021-09-17T01:48:02.077 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : libcephsqlite-2:17.0.0-7730.g2a911e49.el8.x86_64 5/6 2021-09-17T01:48:02.556 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-test-17.0.0-7730.g2a911e49.el8.x86_ 11 MB/s | 43 MB 00:03 2021-09-17T01:48:02.560 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:48:02.560 INFO:teuthology.orchestra.run.smithi045.stdout:Total 12 MB/s | 44 MB 00:03 2021-09-17T01:48:02.564 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:48:02.591 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:48:02.591 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:48:02.846 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:48:02.850 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:48:03.350 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:48:03.535 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2021-09-17T01:48:03.739 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-17T01:48:03.800 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-17T01:48:03.967 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2021-09-17T01:48:04.129 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 4/6 2021-09-17T01:48:04.272 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:17.0.0-7730.g2a911e49.el8.x86_64 5/6 2021-09-17T01:48:06.594 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: libcephsqlite-2:17.0.0-7730.g2a911e49.el8.x86_64 5/6 2021-09-17T01:48:06.907 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-test-2:17.0.0-7730.g2a911e49.el8.x86_64 6/6 2021-09-17T01:48:07.135 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-test-2:17.0.0-7730.g2a911e49.el8.x86_64 6/6 2021-09-17T01:48:07.135 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-test-2:17.0.0-7730.g2a911e49.el8.x86_64 1/6 2021-09-17T01:48:07.136 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : libcephsqlite-2:17.0.0-7730.g2a911e49.el8.x86_64 2/6 2021-09-17T01:48:07.136 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 3/6 2021-09-17T01:48:07.136 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2021-09-17T01:48:07.136 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2021-09-17T01:48:07.472 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2021-09-17T01:48:07.472 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:48:08.787 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:17.0.0-7730.g2a911e49.el8.x86_64 5/6 2021-09-17T01:48:09.044 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:17.0.0-7730.g2a911e49.el8.x86_64 6/6 2021-09-17T01:48:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-test-2:17.0.0-7730.g2a911e49.el8.x86_64 6/6 2021-09-17T01:48:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:17.0.0-7730.g2a911e49.el8.x86_64 1/6 2021-09-17T01:48:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:17.0.0-7730.g2a911e49.el8.x86_64 2/6 2021-09-17T01:48:09.273 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 3/6 2021-09-17T01:48:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2021-09-17T01:48:09.274 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2021-09-17T01:48:09.624 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2021-09-17T01:48:09.624 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:48:18.304 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:47:32 AM UTC. 2021-09-17T01:48:20.582 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:47:35 AM UTC. 2021-09-17T01:48:20.900 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:48:20.900 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:48:20.901 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-test-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:48:20.901 INFO:teuthology.orchestra.run.smithi013.stdout: jq-1.5-12.el8.x86_64 2021-09-17T01:48:20.901 INFO:teuthology.orchestra.run.smithi013.stdout: libcephsqlite-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:48:20.901 INFO:teuthology.orchestra.run.smithi013.stdout: oniguruma-6.8.2-2.el8.x86_64 2021-09-17T01:48:20.901 INFO:teuthology.orchestra.run.smithi013.stdout: socat-1.7.3.3-2.el8.x86_64 2021-09-17T01:48:20.902 INFO:teuthology.orchestra.run.smithi013.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-09-17T01:48:20.902 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:48:20.902 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:48:21.015 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install ceph 2021-09-17T01:48:21.353 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:48:21.875 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-17T01:48:23.237 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:48:23.237 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:48:23.238 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:48:23.238 INFO:teuthology.orchestra.run.smithi045.stdout: jq-1.5-12.el8.x86_64 2021-09-17T01:48:23.238 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:48:23.238 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.8.2-2.el8.x86_64 2021-09-17T01:48:23.239 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.3.3-2.el8.x86_64 2021-09-17T01:48:23.239 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-09-17T01:48:23.239 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:48:23.239 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:48:23.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph 2021-09-17T01:48:23.707 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:48:24.251 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-17T01:48:27.005 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-09-17T01:48:29.504 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 58 kB/s | 2.4 kB 00:00 2021-09-17T01:48:32.887 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:48:35.260 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:48:35.553 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:48:35.556 INFO:teuthology.orchestra.run.smithi013.stdout:======================================================================================================= 2021-09-17T01:48:35.557 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repository Size 2021-09-17T01:48:35.557 INFO:teuthology.orchestra.run.smithi013.stdout:======================================================================================================= 2021-09-17T01:48:35.557 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:48:35.558 INFO:teuthology.orchestra.run.smithi013.stdout: ceph x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 6.4 k 2021-09-17T01:48:35.558 INFO:teuthology.orchestra.run.smithi013.stdout:Installing dependencies: 2021-09-17T01:48:35.558 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mds x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 2.3 M 2021-09-17T01:48:35.559 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 1.5 M 2021-09-17T01:48:35.559 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr-modules-core noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 214 k 2021-09-17T01:48:35.559 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mon x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 3.9 M 2021-09-17T01:48:35.559 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-osd x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 17 M 2021-09-17T01:48:35.560 INFO:teuthology.orchestra.run.smithi013.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-09-17T01:48:35.560 INFO:teuthology.orchestra.run.smithi013.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-09-17T01:48:35.560 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-09-17T01:48:35.561 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-09-17T01:48:35.561 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-09-17T01:48:35.561 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2021-09-17T01:48:35.562 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-09-17T01:48:35.563 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-09-17T01:48:35.563 INFO:teuthology.orchestra.run.smithi013.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-09-17T01:48:35.564 INFO:teuthology.orchestra.run.smithi013.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-09-17T01:48:35.564 INFO:teuthology.orchestra.run.smithi013.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-09-17T01:48:35.564 INFO:teuthology.orchestra.run.smithi013.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-09-17T01:48:35.564 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-09-17T01:48:35.566 INFO:teuthology.orchestra.run.smithi013.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-09-17T01:48:35.566 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-09-17T01:48:35.567 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-09-17T01:48:35.567 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-09-17T01:48:35.567 INFO:teuthology.orchestra.run.smithi013.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-09-17T01:48:35.567 INFO:teuthology.orchestra.run.smithi013.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-09-17T01:48:35.568 INFO:teuthology.orchestra.run.smithi013.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-09-17T01:48:35.568 INFO:teuthology.orchestra.run.smithi013.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-09-17T01:48:35.569 INFO:teuthology.orchestra.run.smithi013.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-09-17T01:48:35.569 INFO:teuthology.orchestra.run.smithi013.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-09-17T01:48:35.569 INFO:teuthology.orchestra.run.smithi013.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-09-17T01:48:35.569 INFO:teuthology.orchestra.run.smithi013.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-09-17T01:48:35.570 INFO:teuthology.orchestra.run.smithi013.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-09-17T01:48:35.570 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:48:35.570 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:48:35.570 INFO:teuthology.orchestra.run.smithi013.stdout:======================================================================================================= 2021-09-17T01:48:35.571 INFO:teuthology.orchestra.run.smithi013.stdout:Install 32 Packages 2021-09-17T01:48:35.571 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:48:35.572 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 28 M 2021-09-17T01:48:35.572 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 112 M 2021-09-17T01:48:35.572 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:48:35.718 INFO:teuthology.orchestra.run.smithi013.stdout:(1/32): ceph-17.0.0-7730.g2a911e49.el8.x86_64.r 42 kB/s | 6.4 kB 00:00 2021-09-17T01:48:35.936 INFO:teuthology.orchestra.run.smithi013.stdout:(2/32): ceph-mgr-17.0.0-7730.g2a911e49.el8.x86_ 4.1 MB/s | 1.5 MB 00:00 2021-09-17T01:48:36.016 INFO:teuthology.orchestra.run.smithi013.stdout:(3/32): ceph-mds-17.0.0-7730.g2a911e49.el8.x86_ 5.0 MB/s | 2.3 MB 00:00 2021-09-17T01:48:36.101 INFO:teuthology.orchestra.run.smithi013.stdout:(4/32): ceph-mgr-modules-core-17.0.0-7730.g2a91 2.5 MB/s | 214 kB 00:00 2021-09-17T01:48:36.234 INFO:teuthology.orchestra.run.smithi013.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 328 kB/s | 44 kB 00:00 2021-09-17T01:48:36.354 INFO:teuthology.orchestra.run.smithi013.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.5 MB/s | 185 kB 00:00 2021-09-17T01:48:36.431 INFO:teuthology.orchestra.run.smithi013.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.2 MB/s | 173 kB 00:00 2021-09-17T01:48:36.601 INFO:teuthology.orchestra.run.smithi013.stdout:(8/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 2.2 MB/s | 384 kB 00:00 2021-09-17T01:48:36.687 INFO:teuthology.orchestra.run.smithi013.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 130 kB/s | 11 kB 00:00 2021-09-17T01:48:36.750 INFO:teuthology.orchestra.run.smithi013.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 277 kB/s | 18 kB 00:00 2021-09-17T01:48:36.823 INFO:teuthology.orchestra.run.smithi013.stdout:(11/32): ceph-mon-17.0.0-7730.g2a911e49.el8.x86 3.5 MB/s | 3.9 MB 00:01 2021-09-17T01:48:36.878 INFO:teuthology.orchestra.run.smithi013.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 382 kB/s | 49 kB 00:00 2021-09-17T01:48:36.952 INFO:teuthology.orchestra.run.smithi013.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 455 kB/s | 59 kB 00:00 2021-09-17T01:48:36.972 INFO:teuthology.orchestra.run.smithi013.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.9 MB/s | 283 kB 00:00 2021-09-17T01:48:36.978 INFO:teuthology.orchestra.run.smithi013.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 609 kB/s | 16 kB 00:00 2021-09-17T01:48:36.997 INFO:teuthology.orchestra.run.smithi013.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 772 kB/s | 19 kB 00:00 2021-09-17T01:48:37.004 INFO:teuthology.orchestra.run.smithi013.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 928 kB/s | 24 kB 00:00 2021-09-17T01:48:37.022 INFO:teuthology.orchestra.run.smithi013.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 1.1 MB/s | 29 kB 00:00 2021-09-17T01:48:37.032 INFO:teuthology.orchestra.run.smithi013.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 997 kB/s | 27 kB 00:00 2021-09-17T01:48:37.098 INFO:teuthology.orchestra.run.smithi013.stdout:(20/32): python3-waitress-1.2.1-2.el8.1.noarch. 3.1 MB/s | 241 kB 00:00 2021-09-17T01:48:37.124 INFO:teuthology.orchestra.run.smithi013.stdout:(21/32): python3-webtest-2.0.33-1.el8.noarch.rp 3.3 MB/s | 85 kB 00:00 2021-09-17T01:48:37.130 INFO:teuthology.orchestra.run.smithi013.stdout:(22/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.5 MB/s | 251 kB 00:00 2021-09-17T01:48:37.153 INFO:teuthology.orchestra.run.smithi013.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 792 kB/s | 23 kB 00:00 2021-09-17T01:48:37.173 INFO:teuthology.orchestra.run.smithi013.stdout:(24/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2021-09-17T01:48:37.204 INFO:teuthology.orchestra.run.smithi013.stdout:(25/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.0 MB/s | 54 kB 00:00 2021-09-17T01:48:37.215 INFO:teuthology.orchestra.run.smithi013.stdout:(26/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 2.4 MB/s | 103 kB 00:00 2021-09-17T01:48:37.234 INFO:teuthology.orchestra.run.smithi013.stdout:(27/32): python3-werkzeug-0.12.2-4.el8.noarch.r 16 MB/s | 457 kB 00:00 2021-09-17T01:48:37.249 INFO:teuthology.orchestra.run.smithi013.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2021-09-17T01:48:37.267 INFO:teuthology.orchestra.run.smithi013.stdout:(29/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 8.7 MB/s | 238 kB 00:00 2021-09-17T01:48:37.283 INFO:teuthology.orchestra.run.smithi013.stdout:(30/32): python3-pycparser-2.14-14.el8.noarch.r 3.2 MB/s | 109 kB 00:00 2021-09-17T01:48:37.300 INFO:teuthology.orchestra.run.smithi013.stdout:(31/32): python3-cryptography-3.2.1-4.el8.x86_6 16 MB/s | 559 kB 00:00 2021-09-17T01:48:37.932 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:48:37.936 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================================= 2021-09-17T01:48:37.936 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-09-17T01:48:37.937 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================================= 2021-09-17T01:48:37.937 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:48:37.937 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 6.4 k 2021-09-17T01:48:37.938 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-09-17T01:48:37.938 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 2.3 M 2021-09-17T01:48:37.938 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 1.5 M 2021-09-17T01:48:37.938 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 214 k 2021-09-17T01:48:37.939 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 3.9 M 2021-09-17T01:48:37.939 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 17 M 2021-09-17T01:48:37.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-09-17T01:48:37.939 INFO:teuthology.orchestra.run.smithi045.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-09-17T01:48:37.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-09-17T01:48:37.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-09-17T01:48:37.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-09-17T01:48:37.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2021-09-17T01:48:37.940 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-09-17T01:48:37.941 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-09-17T01:48:37.941 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-09-17T01:48:37.941 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-09-17T01:48:37.941 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-09-17T01:48:37.941 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-09-17T01:48:37.942 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-09-17T01:48:37.942 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-09-17T01:48:37.942 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-09-17T01:48:37.942 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-09-17T01:48:37.942 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-09-17T01:48:37.943 INFO:teuthology.orchestra.run.smithi045.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-09-17T01:48:37.943 INFO:teuthology.orchestra.run.smithi045.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-09-17T01:48:37.943 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-09-17T01:48:37.943 INFO:teuthology.orchestra.run.smithi045.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-09-17T01:48:37.943 INFO:teuthology.orchestra.run.smithi045.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-09-17T01:48:37.944 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-09-17T01:48:37.944 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-09-17T01:48:37.944 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-09-17T01:48:37.944 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-09-17T01:48:37.944 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:48:37.945 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:48:37.945 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================================= 2021-09-17T01:48:37.945 INFO:teuthology.orchestra.run.smithi045.stdout:Install 32 Packages 2021-09-17T01:48:37.945 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:48:37.946 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 28 M 2021-09-17T01:48:37.946 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 112 M 2021-09-17T01:48:37.947 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:48:38.152 INFO:teuthology.orchestra.run.smithi013.stdout:(32/32): ceph-osd-17.0.0-7730.g2a911e49.el8.x86 7.5 MB/s | 17 MB 00:02 2021-09-17T01:48:38.156 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:48:38.157 INFO:teuthology.orchestra.run.smithi013.stdout:Total 11 MB/s | 28 MB 00:02 2021-09-17T01:48:38.167 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:48:38.211 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:48:38.211 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:48:38.269 INFO:teuthology.orchestra.run.smithi045.stdout:(1/32): ceph-17.0.0-7730.g2a911e49.el8.x86_64.r 20 kB/s | 6.4 kB 00:00 2021-09-17T01:48:38.428 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:48:38.432 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:48:39.090 INFO:teuthology.orchestra.run.smithi045.stdout:(2/32): ceph-mgr-17.0.0-7730.g2a911e49.el8.x86_ 1.3 MB/s | 1.5 MB 00:01 2021-09-17T01:48:39.227 INFO:teuthology.orchestra.run.smithi045.stdout:(3/32): ceph-mds-17.0.0-7730.g2a911e49.el8.x86_ 1.8 MB/s | 2.3 MB 00:01 2021-09-17T01:48:39.275 INFO:teuthology.orchestra.run.smithi045.stdout:(4/32): ceph-mgr-modules-core-17.0.0-7730.g2a91 4.4 MB/s | 214 kB 00:00 2021-09-17T01:48:39.376 INFO:teuthology.orchestra.run.smithi045.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 431 kB/s | 44 kB 00:00 2021-09-17T01:48:39.457 INFO:teuthology.orchestra.run.smithi045.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 2.3 MB/s | 185 kB 00:00 2021-09-17T01:48:39.495 INFO:teuthology.orchestra.run.smithi045.stdout:(7/32): ceph-mon-17.0.0-7730.g2a911e49.el8.x86_ 3.2 MB/s | 3.9 MB 00:01 2021-09-17T01:48:39.511 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:48:39.512 INFO:teuthology.orchestra.run.smithi045.stdout:(8/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 3.1 MB/s | 173 kB 00:00 2021-09-17T01:48:39.539 INFO:teuthology.orchestra.run.smithi045.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 405 kB/s | 11 kB 00:00 2021-09-17T01:48:39.565 INFO:teuthology.orchestra.run.smithi045.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 685 kB/s | 18 kB 00:00 2021-09-17T01:48:39.595 INFO:teuthology.orchestra.run.smithi045.stdout:(11/32): python3-logutils-0.3.5-11.el8.noarch.r 1.6 MB/s | 49 kB 00:00 2021-09-17T01:48:39.624 INFO:teuthology.orchestra.run.smithi045.stdout:(12/32): python3-more-itertools-7.2.0-3.el8.noa 2.0 MB/s | 59 kB 00:00 2021-09-17T01:48:39.728 INFO:teuthology.orchestra.run.smithi045.stdout:(13/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2021-09-17T01:48:39.735 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-09-17T01:48:39.745 INFO:teuthology.orchestra.run.smithi045.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.3 MB/s | 283 kB 00:00 2021-09-17T01:48:39.758 INFO:teuthology.orchestra.run.smithi045.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 523 kB/s | 16 kB 00:00 2021-09-17T01:48:39.775 INFO:teuthology.orchestra.run.smithi045.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 615 kB/s | 19 kB 00:00 2021-09-17T01:48:39.789 INFO:teuthology.orchestra.run.smithi045.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 771 kB/s | 24 kB 00:00 2021-09-17T01:48:39.805 INFO:teuthology.orchestra.run.smithi045.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 998 kB/s | 29 kB 00:00 2021-09-17T01:48:39.819 INFO:teuthology.orchestra.run.smithi045.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 937 kB/s | 27 kB 00:00 2021-09-17T01:48:39.913 INFO:teuthology.orchestra.run.smithi045.stdout:(20/32): python3-waitress-1.2.1-2.el8.1.noarch. 2.2 MB/s | 241 kB 00:00 2021-09-17T01:48:39.927 INFO:teuthology.orchestra.run.smithi045.stdout:(21/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.3 MB/s | 251 kB 00:00 2021-09-17T01:48:39.950 INFO:teuthology.orchestra.run.smithi045.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.3 MB/s | 85 kB 00:00 2021-09-17T01:48:39.963 INFO:teuthology.orchestra.run.smithi045.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 635 kB/s | 23 kB 00:00 2021-09-17T01:48:39.997 INFO:teuthology.orchestra.run.smithi045.stdout:(24/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.2 MB/s | 157 kB 00:00 2021-09-17T01:48:40.013 INFO:teuthology.orchestra.run.smithi045.stdout:(25/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2021-09-17T01:48:40.030 INFO:teuthology.orchestra.run.smithi045.stdout:(26/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 3.2 MB/s | 103 kB 00:00 2021-09-17T01:48:40.055 INFO:teuthology.orchestra.run.smithi045.stdout:(27/32): python3-werkzeug-0.12.2-4.el8.noarch.r 11 MB/s | 457 kB 00:00 2021-09-17T01:48:40.066 INFO:teuthology.orchestra.run.smithi045.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2021-09-17T01:48:40.083 INFO:teuthology.orchestra.run.smithi045.stdout:(29/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 8.6 MB/s | 238 kB 00:00 2021-09-17T01:48:40.100 INFO:teuthology.orchestra.run.smithi045.stdout:(30/32): python3-pycparser-2.14-14.el8.noarch.r 3.1 MB/s | 109 kB 00:00 2021-09-17T01:48:40.106 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-09-17T01:48:40.118 INFO:teuthology.orchestra.run.smithi045.stdout:(31/32): python3-cryptography-3.2.1-4.el8.x86_6 19 MB/s | 559 kB 00:00 2021-09-17T01:48:40.315 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2021-09-17T01:48:40.430 INFO:teuthology.orchestra.run.smithi045.stdout:(32/32): ceph-osd-17.0.0-7730.g2a911e49.el8.x86 12 MB/s | 17 MB 00:01 2021-09-17T01:48:40.434 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:48:40.434 INFO:teuthology.orchestra.run.smithi045.stdout:Total 11 MB/s | 28 MB 00:02 2021-09-17T01:48:40.443 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:48:40.482 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:48:40.488 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:48:40.689 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 4/32 2021-09-17T01:48:40.696 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:48:40.701 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:48:40.911 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 5/32 2021-09-17T01:48:41.064 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2021-09-17T01:48:41.208 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2021-09-17T01:48:41.386 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2021-09-17T01:48:41.591 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 9/32 2021-09-17T01:48:41.863 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 10/32 2021-09-17T01:48:41.999 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:48:42.071 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-09-17T01:48:42.238 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-09-17T01:48:42.265 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 12/32 2021-09-17T01:48:42.489 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2021-09-17T01:48:42.492 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-09-17T01:48:42.671 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 14/32 2021-09-17T01:48:42.752 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2021-09-17T01:48:42.852 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2021-09-17T01:48:43.036 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2021-09-17T01:48:43.279 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 4/32 2021-09-17T01:48:43.326 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2021-09-17T01:48:43.463 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2021-09-17T01:48:43.564 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 5/32 2021-09-17T01:48:43.688 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2021-09-17T01:48:43.709 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2021-09-17T01:48:43.878 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2021-09-17T01:48:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2021-09-17T01:48:44.075 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-09-17T01:48:44.124 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2021-09-17T01:48:44.310 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-09-17T01:48:44.340 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 9/32 2021-09-17T01:48:44.534 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-09-17T01:48:44.631 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 10/32 2021-09-17T01:48:44.754 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2021-09-17T01:48:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-09-17T01:48:45.034 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 12/32 2021-09-17T01:48:45.111 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2021-09-17T01:48:45.307 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2021-09-17T01:48:45.463 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-09-17T01:48:45.532 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 14/32 2021-09-17T01:48:45.747 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2021-09-17T01:48:45.803 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-7730.g2a911e49.el8. 27/32 2021-09-17T01:48:45.906 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2021-09-17T01:48:45.966 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-mgr-2:17.0.0-7730.g2a911e49.el8.x86_64 28/32 2021-09-17T01:48:46.104 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2021-09-17T01:48:46.232 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2021-09-17T01:48:46.499 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2021-09-17T01:48:46.835 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2021-09-17T01:48:47.074 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-09-17T01:48:47.346 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-09-17T01:48:47.550 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-mgr-2:17.0.0-7730.g2a911e49.el8.x86_64 28/32 2021-09-17T01:48:47.611 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-09-17T01:48:47.758 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-osd-2:17.0.0-7730.g2a911e49.el8.x86_64 29/32 2021-09-17T01:48:47.839 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2021-09-17T01:48:48.162 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2021-09-17T01:48:48.173 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-osd-2:17.0.0-7730.g2a911e49.el8.x86_64 29/32 2021-09-17T01:48:48.346 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-mon-2:17.0.0-7730.g2a911e49.el8.x86_64 30/32 2021-09-17T01:48:48.522 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-09-17T01:48:48.577 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-mon-2:17.0.0-7730.g2a911e49.el8.x86_64 30/32 2021-09-17T01:48:48.747 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-mds-2:17.0.0-7730.g2a911e49.el8.x86_64 31/32 2021-09-17T01:48:48.765 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-mds-2:17.0.0-7730.g2a911e49.el8.x86_64 31/32 2021-09-17T01:48:48.869 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-2:17.0.0-7730.g2a911e49.el8.x86_64 32/32 2021-09-17T01:48:48.879 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-7730.g2a911e49.el8. 27/32 2021-09-17T01:48:49.026 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:17.0.0-7730.g2a911e49.el8.x86_64 28/32 2021-09-17T01:48:49.232 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-2:17.0.0-7730.g2a911e49.el8.x86_64 32/32 2021-09-17T01:48:49.233 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-2:17.0.0-7730.g2a911e49.el8.x86_64 1/32 2021-09-17T01:48:49.233 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-mds-2:17.0.0-7730.g2a911e49.el8.x86_64 2/32 2021-09-17T01:48:49.233 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-mgr-2:17.0.0-7730.g2a911e49.el8.x86_64 3/32 2021-09-17T01:48:49.233 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-mon-2:17.0.0-7730.g2a911e49.el8.x86_64 4/32 2021-09-17T01:48:49.234 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-osd-2:17.0.0-7730.g2a911e49.el8.x86_64 5/32 2021-09-17T01:48:49.234 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-7730.g2a911e49.el8. 6/32 2021-09-17T01:48:49.234 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2021-09-17T01:48:49.234 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2021-09-17T01:48:49.235 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2021-09-17T01:48:49.236 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2021-09-17T01:48:49.236 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2021-09-17T01:48:49.236 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2021-09-17T01:48:49.236 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2021-09-17T01:48:49.237 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2021-09-17T01:48:49.237 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2021-09-17T01:48:49.237 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2021-09-17T01:48:49.237 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2021-09-17T01:48:49.237 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2021-09-17T01:48:49.238 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2021-09-17T01:48:49.238 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2021-09-17T01:48:49.238 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 21/32 2021-09-17T01:48:49.239 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2021-09-17T01:48:49.239 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2021-09-17T01:48:49.239 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2021-09-17T01:48:49.239 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 25/32 2021-09-17T01:48:49.240 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 26/32 2021-09-17T01:48:49.241 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2021-09-17T01:48:49.243 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2021-09-17T01:48:49.243 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2021-09-17T01:48:49.244 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 30/32 2021-09-17T01:48:49.244 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 31/32 2021-09-17T01:48:49.918 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 32/32 2021-09-17T01:48:49.918 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:48:50.617 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:17.0.0-7730.g2a911e49.el8.x86_64 28/32 2021-09-17T01:48:50.818 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:17.0.0-7730.g2a911e49.el8.x86_64 29/32 2021-09-17T01:48:51.234 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:17.0.0-7730.g2a911e49.el8.x86_64 29/32 2021-09-17T01:48:51.441 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:17.0.0-7730.g2a911e49.el8.x86_64 30/32 2021-09-17T01:48:51.671 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:17.0.0-7730.g2a911e49.el8.x86_64 30/32 2021-09-17T01:48:51.800 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:17.0.0-7730.g2a911e49.el8.x86_64 31/32 2021-09-17T01:48:51.818 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:17.0.0-7730.g2a911e49.el8.x86_64 31/32 2021-09-17T01:48:51.955 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:17.0.0-7730.g2a911e49.el8.x86_64 32/32 2021-09-17T01:48:52.320 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-2:17.0.0-7730.g2a911e49.el8.x86_64 32/32 2021-09-17T01:48:52.321 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:17.0.0-7730.g2a911e49.el8.x86_64 1/32 2021-09-17T01:48:52.321 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:17.0.0-7730.g2a911e49.el8.x86_64 2/32 2021-09-17T01:48:52.321 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:17.0.0-7730.g2a911e49.el8.x86_64 3/32 2021-09-17T01:48:52.321 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:17.0.0-7730.g2a911e49.el8.x86_64 4/32 2021-09-17T01:48:52.322 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:17.0.0-7730.g2a911e49.el8.x86_64 5/32 2021-09-17T01:48:52.322 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-7730.g2a911e49.el8. 6/32 2021-09-17T01:48:52.322 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2021-09-17T01:48:52.322 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2021-09-17T01:48:52.323 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2021-09-17T01:48:52.323 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2021-09-17T01:48:52.323 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2021-09-17T01:48:52.323 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2021-09-17T01:48:52.323 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2021-09-17T01:48:52.324 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2021-09-17T01:48:52.324 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2021-09-17T01:48:52.324 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2021-09-17T01:48:52.324 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2021-09-17T01:48:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2021-09-17T01:48:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2021-09-17T01:48:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2021-09-17T01:48:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 21/32 2021-09-17T01:48:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2021-09-17T01:48:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2021-09-17T01:48:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2021-09-17T01:48:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 25/32 2021-09-17T01:48:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 26/32 2021-09-17T01:48:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2021-09-17T01:48:52.327 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2021-09-17T01:48:52.328 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2021-09-17T01:48:52.328 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 30/32 2021-09-17T01:48:52.328 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 31/32 2021-09-17T01:48:53.088 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 32/32 2021-09-17T01:48:53.089 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:49:01.083 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:48:12 AM UTC. 2021-09-17T01:49:03.713 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:49:03.714 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:49:03.714 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:49:03.714 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mds-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:49:03.714 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:49:03.715 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr-modules-core-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:49:03.715 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mon-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:49:03.715 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-osd-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:49:03.715 INFO:teuthology.orchestra.run.smithi013.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-09-17T01:49:03.716 INFO:teuthology.orchestra.run.smithi013.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-09-17T01:49:03.716 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-09-17T01:49:03.716 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-09-17T01:49:03.716 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-09-17T01:49:03.717 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-09-17T01:49:03.717 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jaraco-6.2-6.el8.noarch 2021-09-17T01:49:03.717 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-09-17T01:49:03.717 INFO:teuthology.orchestra.run.smithi013.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-09-17T01:49:03.717 INFO:teuthology.orchestra.run.smithi013.stdout: python3-mako-1.0.6-13.el8.noarch 2021-09-17T01:49:03.718 INFO:teuthology.orchestra.run.smithi013.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-09-17T01:49:03.718 INFO:teuthology.orchestra.run.smithi013.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-09-17T01:49:03.718 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-09-17T01:49:03.718 INFO:teuthology.orchestra.run.smithi013.stdout: python3-portend-2.6-1.el8.noarch 2021-09-17T01:49:03.719 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-09-17T01:49:03.719 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pycparser-2.14-14.el8.noarch 2021-09-17T01:49:03.719 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pytz-2017.2-9.el8.noarch 2021-09-17T01:49:03.719 INFO:teuthology.orchestra.run.smithi013.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-09-17T01:49:03.720 INFO:teuthology.orchestra.run.smithi013.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-09-17T01:49:03.720 INFO:teuthology.orchestra.run.smithi013.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-09-17T01:49:03.720 INFO:teuthology.orchestra.run.smithi013.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-09-17T01:49:03.720 INFO:teuthology.orchestra.run.smithi013.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-09-17T01:49:03.721 INFO:teuthology.orchestra.run.smithi013.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-09-17T01:49:03.721 INFO:teuthology.orchestra.run.smithi013.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-09-17T01:49:03.721 INFO:teuthology.orchestra.run.smithi013.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-09-17T01:49:03.721 INFO:teuthology.orchestra.run.smithi013.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-09-17T01:49:03.721 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:49:03.722 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:49:03.843 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install ceph-base 2021-09-17T01:49:04.187 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:48:15 AM UTC. 2021-09-17T01:49:04.193 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:49:04.762 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 71 kB/s | 2.8 kB 00:00 2021-09-17T01:49:06.862 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:49:06.863 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:49:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:49:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:49:06.863 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:49:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:49:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:49:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:49:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-09-17T01:49:06.864 INFO:teuthology.orchestra.run.smithi045.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-09-17T01:49:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-09-17T01:49:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-09-17T01:49:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-09-17T01:49:06.865 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-09-17T01:49:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-6.2-6.el8.noarch 2021-09-17T01:49:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-09-17T01:49:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-09-17T01:49:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.0.6-13.el8.noarch 2021-09-17T01:49:06.866 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-09-17T01:49:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-09-17T01:49:06.867 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-09-17T01:49:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-2.6-1.el8.noarch 2021-09-17T01:49:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-09-17T01:49:06.868 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.14-14.el8.noarch 2021-09-17T01:49:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2017.2-9.el8.noarch 2021-09-17T01:49:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-09-17T01:49:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-09-17T01:49:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-09-17T01:49:06.869 INFO:teuthology.orchestra.run.smithi045.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-09-17T01:49:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-09-17T01:49:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-09-17T01:49:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-09-17T01:49:06.870 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-09-17T01:49:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-09-17T01:49:06.871 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:49:06.871 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:49:06.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-base 2021-09-17T01:49:07.346 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:49:07.918 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-09-17T01:49:10.162 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-17T01:49:13.116 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-17T01:49:15.880 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:49:18.481 INFO:teuthology.orchestra.run.smithi013.stdout:Package ceph-base-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:49:18.719 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:49:18.932 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:49:18.933 INFO:teuthology.orchestra.run.smithi013.stdout:Nothing to do. 2021-09-17T01:49:18.933 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:49:19.028 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install cephadm 2021-09-17T01:49:19.378 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:49:19.918 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-17T01:49:20.112 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-17T01:49:21.436 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-base-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:49:21.880 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:49:21.881 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-09-17T01:49:21.882 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:49:21.992 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install cephadm 2021-09-17T01:49:22.070 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:49:22.071 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:49:22.071 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repository Size 2021-09-17T01:49:22.072 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:49:22.072 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:49:22.072 INFO:teuthology.orchestra.run.smithi013.stdout: cephadm noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 70 k 2021-09-17T01:49:22.073 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:49:22.073 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:49:22.073 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:49:22.073 INFO:teuthology.orchestra.run.smithi013.stdout:Install 1 Package 2021-09-17T01:49:22.073 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:49:22.074 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 70 k 2021-09-17T01:49:22.074 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 310 k 2021-09-17T01:49:22.074 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:49:22.203 INFO:teuthology.orchestra.run.smithi013.stdout:cephadm-17.0.0-7730.g2a911e49.el8.noarch.rpm 529 kB/s | 70 kB 00:00 2021-09-17T01:49:22.204 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:49:22.205 INFO:teuthology.orchestra.run.smithi013.stdout:Total 523 kB/s | 70 kB 00:00 2021-09-17T01:49:22.205 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:49:22.212 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:49:22.213 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:49:22.214 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:49:22.219 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:49:22.268 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:49:22.340 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:49:22.887 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-17T01:49:23.076 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-17T01:49:24.706 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 1/1 2021-09-17T01:49:24.844 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 1/1 2021-09-17T01:49:24.950 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:49:24.951 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:49:24.951 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-09-17T01:49:24.952 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:49:24.952 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:49:24.952 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 70 k 2021-09-17T01:49:24.952 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:49:24.953 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:49:24.953 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:49:24.953 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-09-17T01:49:24.954 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:49:24.954 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 70 k 2021-09-17T01:49:24.954 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 310 k 2021-09-17T01:49:24.954 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:49:25.012 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 1/1 2021-09-17T01:49:25.085 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm-17.0.0-7730.g2a911e49.el8.noarch.rpm 524 kB/s | 70 kB 00:00 2021-09-17T01:49:25.085 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:49:25.086 INFO:teuthology.orchestra.run.smithi045.stdout:Total 518 kB/s | 70 kB 00:00 2021-09-17T01:49:25.086 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:49:25.093 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:49:25.093 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:49:25.095 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:49:25.099 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:49:25.147 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:49:25.257 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 1/1 2021-09-17T01:49:25.257 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:49:27.596 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 1/1 2021-09-17T01:49:27.824 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 1/1 2021-09-17T01:49:27.997 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 1/1 2021-09-17T01:49:28.222 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 1/1 2021-09-17T01:49:28.223 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:49:36.201 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:48:55 AM UTC. 2021-09-17T01:49:38.830 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:49:38.830 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:49:38.830 INFO:teuthology.orchestra.run.smithi013.stdout: cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:49:38.831 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:49:38.831 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:49:38.955 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install ceph-immutable-object-cache 2021-09-17T01:49:39.315 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:49:39.743 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:48:58 AM UTC. 2021-09-17T01:49:39.881 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-17T01:49:42.346 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:49:42.347 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:49:42.347 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:49:42.347 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:49:42.348 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:49:42.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-immutable-object-cache 2021-09-17T01:49:42.933 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:49:43.481 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:49:45.065 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-17T01:49:48.582 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-17T01:49:50.709 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:49:53.409 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:49:53.410 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:49:53.411 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repo Size 2021-09-17T01:49:53.411 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:49:53.411 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:49:53.411 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 169 k 2021-09-17T01:49:53.412 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:49:53.412 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:49:53.412 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:49:53.412 INFO:teuthology.orchestra.run.smithi013.stdout:Install 1 Package 2021-09-17T01:49:53.413 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:49:53.413 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 169 k 2021-09-17T01:49:53.413 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 567 k 2021-09-17T01:49:53.413 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:49:53.565 INFO:teuthology.orchestra.run.smithi013.stdout:ceph-immutable-object-cache-17.0.0-7730.g2a911e 1.1 MB/s | 169 kB 00:00 2021-09-17T01:49:53.565 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:49:53.566 INFO:teuthology.orchestra.run.smithi013.stdout:Total 1.1 MB/s | 169 kB 00:00 2021-09-17T01:49:53.566 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:49:53.579 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:49:53.580 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:49:53.637 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:49:53.641 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:49:53.764 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:49:53.905 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-7730.g2a911e49. 1/1 2021-09-17T01:49:54.137 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:49:54.272 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-7730.g2a911e49. 1/1 2021-09-17T01:49:54.537 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-7730.g2a911e49. 1/1 2021-09-17T01:49:54.537 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:49:56.865 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:49:56.866 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:49:56.866 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2021-09-17T01:49:56.866 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:49:56.867 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:49:56.867 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 169 k 2021-09-17T01:49:56.867 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:49:56.867 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:49:56.867 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:49:56.868 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-09-17T01:49:56.868 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:49:56.868 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 169 k 2021-09-17T01:49:56.868 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 567 k 2021-09-17T01:49:56.868 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:49:57.023 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-immutable-object-cache-17.0.0-7730.g2a911e 1.1 MB/s | 169 kB 00:00 2021-09-17T01:49:57.024 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:49:57.024 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.0 MB/s | 169 kB 00:00 2021-09-17T01:49:57.024 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:49:57.038 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:49:57.038 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:49:57.095 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:49:57.100 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:49:57.233 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:49:57.398 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-7730.g2a911e49. 1/1 2021-09-17T01:49:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-7730.g2a911e49. 1/1 2021-09-17T01:49:58.011 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-7730.g2a911e49. 1/1 2021-09-17T01:49:58.011 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:50:05.600 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:49:30 AM UTC. 2021-09-17T01:50:08.318 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:50:08.319 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:50:08.319 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-immutable-object-cache-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:50:08.319 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:50:08.319 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:50:08.467 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install ceph-mgr 2021-09-17T01:50:08.819 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:50:08.961 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:49:33 AM UTC. 2021-09-17T01:50:09.380 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-17T01:50:11.795 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:50:11.796 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:50:11.796 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:50:11.796 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:50:11.796 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:50:11.974 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr 2021-09-17T01:50:12.325 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:50:12.902 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 65 kB/s | 2.8 kB 00:00 2021-09-17T01:50:14.494 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 51 kB/s | 2.4 kB 00:00 2021-09-17T01:50:18.193 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-17T01:50:20.109 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:50:22.312 INFO:teuthology.orchestra.run.smithi013.stdout:Package ceph-mgr-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:50:22.760 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:50:22.760 INFO:teuthology.orchestra.run.smithi013.stdout:Nothing to do. 2021-09-17T01:50:22.761 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:50:22.855 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install ceph-mgr-dashboard 2021-09-17T01:50:23.202 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:50:23.758 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-09-17T01:50:23.845 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:50:23.950 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-17T01:50:25.847 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:50:25.848 INFO:teuthology.orchestra.run.smithi013.stdout:=================================================================================================== 2021-09-17T01:50:25.848 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repository Size 2021-09-17T01:50:25.849 INFO:teuthology.orchestra.run.smithi013.stdout:=================================================================================================== 2021-09-17T01:50:25.849 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:50:25.849 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr-dashboard noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 1.5 M 2021-09-17T01:50:25.850 INFO:teuthology.orchestra.run.smithi013.stdout:Installing dependencies: 2021-09-17T01:50:25.850 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-grafana-dashboards noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 24 k 2021-09-17T01:50:25.850 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-prometheus-alerts noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 8.9 k 2021-09-17T01:50:25.850 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-17T01:50:25.851 INFO:teuthology.orchestra.run.smithi013.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-09-17T01:50:25.851 INFO:teuthology.orchestra.run.smithi013.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-09-17T01:50:25.852 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:50:25.852 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:50:25.853 INFO:teuthology.orchestra.run.smithi013.stdout:=================================================================================================== 2021-09-17T01:50:25.853 INFO:teuthology.orchestra.run.smithi013.stdout:Install 6 Packages 2021-09-17T01:50:25.853 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:50:25.854 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 1.8 M 2021-09-17T01:50:25.854 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 7.4 M 2021-09-17T01:50:25.854 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:50:25.954 INFO:teuthology.orchestra.run.smithi013.stdout:(1/6): ceph-prometheus-alerts-17.0.0-7730.g2a91 86 kB/s | 8.9 kB 00:00 2021-09-17T01:50:25.962 INFO:teuthology.orchestra.run.smithi013.stdout:(2/6): ceph-grafana-dashboards-17.0.0-7730.g2a9 214 kB/s | 24 kB 00:00 2021-09-17T01:50:26.034 INFO:teuthology.orchestra.run.smithi013.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 423 kB/s | 34 kB 00:00 2021-09-17T01:50:26.073 INFO:teuthology.orchestra.run.smithi013.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.1 MB/s | 43 kB 00:00 2021-09-17T01:50:26.108 INFO:teuthology.orchestra.run.smithi013.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.3 MB/s | 196 kB 00:00 2021-09-17T01:50:26.193 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-mgr-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:50:26.295 INFO:teuthology.orchestra.run.smithi013.stdout:(6/6): ceph-mgr-dashboard-17.0.0-7730.g2a911e49 3.5 MB/s | 1.5 MB 00:00 2021-09-17T01:50:26.298 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:50:26.299 INFO:teuthology.orchestra.run.smithi013.stdout:Total 4.1 MB/s | 1.8 MB 00:00 2021-09-17T01:50:26.299 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:50:26.308 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:50:26.308 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:50:26.366 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:50:26.370 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:50:26.639 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:50:26.665 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:50:26.666 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-09-17T01:50:26.666 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:50:26.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-dashboard 2021-09-17T01:50:26.846 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-09-17T01:50:27.020 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-09-17T01:50:27.124 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:50:27.186 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-09-17T01:50:27.308 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-7730.g2a911e49.el8.n 4/6 2021-09-17T01:50:27.651 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-7730.g2a911e49.el8. 5/6 2021-09-17T01:50:27.679 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-17T01:50:27.861 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-7730.g2a911e49.el8.noarc 6/6 2021-09-17T01:50:27.873 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-17T01:50:28.076 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-7730.g2a911e49.el8.noarc 6/6 2021-09-17T01:50:28.076 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-7730.g2a911e49.el8. 1/6 2021-09-17T01:50:28.077 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-7730.g2a911e49.el8.noarc 2/6 2021-09-17T01:50:28.078 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-7730.g2a911e49.el8.n 3/6 2021-09-17T01:50:28.078 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-09-17T01:50:28.078 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-09-17T01:50:28.353 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-09-17T01:50:28.354 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:50:29.851 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:50:29.851 INFO:teuthology.orchestra.run.smithi045.stdout:=================================================================================================== 2021-09-17T01:50:29.852 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-09-17T01:50:29.852 INFO:teuthology.orchestra.run.smithi045.stdout:=================================================================================================== 2021-09-17T01:50:29.852 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:50:29.852 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 1.5 M 2021-09-17T01:50:29.853 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-09-17T01:50:29.853 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 24 k 2021-09-17T01:50:29.853 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 8.9 k 2021-09-17T01:50:29.853 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-17T01:50:29.853 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-09-17T01:50:29.854 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-09-17T01:50:29.854 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:50:29.854 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:50:29.854 INFO:teuthology.orchestra.run.smithi045.stdout:=================================================================================================== 2021-09-17T01:50:29.855 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2021-09-17T01:50:29.855 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:50:29.856 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 1.8 M 2021-09-17T01:50:29.856 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 7.4 M 2021-09-17T01:50:29.863 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:50:29.965 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): ceph-prometheus-alerts-17.0.0-7730.g2a91 80 kB/s | 8.9 kB 00:00 2021-09-17T01:50:29.978 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): ceph-grafana-dashboards-17.0.0-7730.g2a9 192 kB/s | 24 kB 00:00 2021-09-17T01:50:30.039 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 451 kB/s | 34 kB 00:00 2021-09-17T01:50:30.081 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2021-09-17T01:50:30.117 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2021-09-17T01:50:30.194 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-mgr-dashboard-17.0.0-7730.g2a911e49 4.5 MB/s | 1.5 MB 00:00 2021-09-17T01:50:30.197 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:50:30.197 INFO:teuthology.orchestra.run.smithi045.stdout:Total 5.3 MB/s | 1.8 MB 00:00 2021-09-17T01:50:30.198 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:50:30.208 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:50:30.208 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:50:30.265 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:50:30.269 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:50:30.560 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:50:30.764 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-09-17T01:50:30.980 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-09-17T01:50:31.170 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-09-17T01:50:31.293 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-7730.g2a911e49.el8.n 4/6 2021-09-17T01:50:31.798 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-7730.g2a911e49.el8. 5/6 2021-09-17T01:50:32.027 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-7730.g2a911e49.el8.noarc 6/6 2021-09-17T01:50:32.259 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-7730.g2a911e49.el8.noarc 6/6 2021-09-17T01:50:32.260 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-7730.g2a911e49.el8. 1/6 2021-09-17T01:50:32.260 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-7730.g2a911e49.el8.noarc 2/6 2021-09-17T01:50:32.260 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-7730.g2a911e49.el8.n 3/6 2021-09-17T01:50:32.260 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-09-17T01:50:32.261 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-09-17T01:50:32.538 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-09-17T01:50:32.539 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:50:39.451 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:50:00 AM UTC. 2021-09-17T01:50:42.037 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:50:42.038 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:50:42.038 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-grafana-dashboards-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:50:42.038 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr-dashboard-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:50:42.038 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-prometheus-alerts-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:50:42.039 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-09-17T01:50:42.039 INFO:teuthology.orchestra.run.smithi013.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-09-17T01:50:42.039 INFO:teuthology.orchestra.run.smithi013.stdout: python3-routes-2.4.1-12.el8.noarch 2021-09-17T01:50:42.039 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:50:42.040 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:50:42.191 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install ceph-mgr-diskprediction-local 2021-09-17T01:50:42.542 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:50:43.093 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-17T01:50:43.522 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:50:03 AM UTC. 2021-09-17T01:50:46.181 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:50:46.182 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:50:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:50:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:50:46.182 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:50:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-09-17T01:50:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-09-17T01:50:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.4.1-12.el8.noarch 2021-09-17T01:50:46.183 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:50:46.183 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:50:46.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-diskprediction-local 2021-09-17T01:50:46.663 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:50:47.218 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:50:48.200 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-09-17T01:50:52.515 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 58 kB/s | 2.4 kB 00:00 2021-09-17T01:50:53.744 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:50:56.431 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:50:56.433 INFO:teuthology.orchestra.run.smithi013.stdout:====================================================================================================================== 2021-09-17T01:50:56.433 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repository Size 2021-09-17T01:50:56.433 INFO:teuthology.orchestra.run.smithi013.stdout:====================================================================================================================== 2021-09-17T01:50:56.434 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:50:56.434 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 7.4 M 2021-09-17T01:50:56.434 INFO:teuthology.orchestra.run.smithi013.stdout:Installing dependencies: 2021-09-17T01:50:56.434 INFO:teuthology.orchestra.run.smithi013.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-09-17T01:50:56.435 INFO:teuthology.orchestra.run.smithi013.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-09-17T01:50:56.435 INFO:teuthology.orchestra.run.smithi013.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2021-09-17T01:50:56.435 INFO:teuthology.orchestra.run.smithi013.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2021-09-17T01:50:56.435 INFO:teuthology.orchestra.run.smithi013.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-09-17T01:50:56.435 INFO:teuthology.orchestra.run.smithi013.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-09-17T01:50:56.436 INFO:teuthology.orchestra.run.smithi013.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-09-17T01:50:56.436 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:50:56.436 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:50:56.436 INFO:teuthology.orchestra.run.smithi013.stdout:====================================================================================================================== 2021-09-17T01:50:56.437 INFO:teuthology.orchestra.run.smithi013.stdout:Install 8 Packages 2021-09-17T01:50:56.437 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:50:56.438 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 36 M 2021-09-17T01:50:56.438 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 199 M 2021-09-17T01:50:56.438 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:50:56.532 INFO:teuthology.orchestra.run.smithi013.stdout:(1/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 39 MB/s | 3.7 MB 00:00 2021-09-17T01:50:56.604 INFO:teuthology.orchestra.run.smithi013.stdout:(2/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 66 MB/s | 4.7 MB 00:00 2021-09-17T01:50:56.746 INFO:teuthology.orchestra.run.smithi013.stdout:(3/8): python3-scipy-1.0.0-20.module+el8.1.0+33 47 MB/s | 15 MB 00:00 2021-09-17T01:50:56.774 INFO:teuthology.orchestra.run.smithi013.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 8.1 MB/s | 225 kB 00:00 2021-09-17T01:50:56.798 INFO:teuthology.orchestra.run.smithi013.stdout:(5/8): openblas-0.3.12-1.el8.x86_64.rpm 24 MB/s | 4.6 MB 00:00 2021-09-17T01:50:56.818 INFO:teuthology.orchestra.run.smithi013.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 3.7 MB/s | 169 kB 00:00 2021-09-17T01:50:56.838 INFO:teuthology.orchestra.run.smithi013.stdout:(7/8): libgfortran-8.4.1-1.el8.x86_64.rpm 19 MB/s | 639 kB 00:00 2021-09-17T01:50:57.349 INFO:teuthology.orchestra.run.smithi013.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-773 8.1 MB/s | 7.4 MB 00:00 2021-09-17T01:50:57.351 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:50:57.351 INFO:teuthology.orchestra.run.smithi013.stdout:Total 39 MB/s | 36 MB 00:00 2021-09-17T01:50:57.361 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:50:57.381 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:50:57.382 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:50:57.695 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:50:57.699 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:50:58.119 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:50:58.238 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:50:58.396 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-17T01:50:58.530 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-17T01:50:58.699 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-17T01:50:59.261 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-17T01:50:59.421 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-17T01:51:00.001 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-17T01:51:00.171 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2021-09-17T01:51:00.865 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:51:00.866 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================================================== 2021-09-17T01:51:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-09-17T01:51:00.867 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================================================== 2021-09-17T01:51:00.867 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:51:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 7.4 M 2021-09-17T01:51:00.868 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-09-17T01:51:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-09-17T01:51:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-09-17T01:51:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2021-09-17T01:51:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2021-09-17T01:51:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-09-17T01:51:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-09-17T01:51:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-09-17T01:51:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:51:00.870 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:51:00.870 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================================================== 2021-09-17T01:51:00.871 INFO:teuthology.orchestra.run.smithi045.stdout:Install 8 Packages 2021-09-17T01:51:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:51:00.872 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 36 M 2021-09-17T01:51:00.872 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 199 M 2021-09-17T01:51:00.872 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:51:00.926 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2021-09-17T01:51:00.967 INFO:teuthology.orchestra.run.smithi045.stdout:(1/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 38 MB/s | 3.7 MB 00:00 2021-09-17T01:51:01.034 INFO:teuthology.orchestra.run.smithi045.stdout:(2/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 71 MB/s | 4.7 MB 00:00 2021-09-17T01:51:01.166 INFO:teuthology.orchestra.run.smithi045.stdout:(3/8): python3-scipy-1.0.0-20.module+el8.1.0+33 49 MB/s | 15 MB 00:00 2021-09-17T01:51:01.195 INFO:teuthology.orchestra.run.smithi045.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 7.8 MB/s | 225 kB 00:00 2021-09-17T01:51:01.227 INFO:teuthology.orchestra.run.smithi045.stdout:(5/8): openblas-0.3.12-1.el8.x86_64.rpm 24 MB/s | 4.6 MB 00:00 2021-09-17T01:51:01.229 INFO:teuthology.orchestra.run.smithi045.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 4.8 MB/s | 169 kB 00:00 2021-09-17T01:51:01.235 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-09-17T01:51:01.257 INFO:teuthology.orchestra.run.smithi045.stdout:(7/8): libgfortran-8.4.1-1.el8.x86_64.rpm 21 MB/s | 639 kB 00:00 2021-09-17T01:51:01.454 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-17T01:51:01.880 INFO:teuthology.orchestra.run.smithi045.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-773 7.3 MB/s | 7.4 MB 00:01 2021-09-17T01:51:01.881 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:51:01.881 INFO:teuthology.orchestra.run.smithi045.stdout:Total 36 MB/s | 36 MB 00:01 2021-09-17T01:51:01.891 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:51:01.912 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:51:01.913 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:51:02.226 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:51:02.230 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:51:02.960 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:51:03.146 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-17T01:51:03.282 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-17T01:51:03.487 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-17T01:51:03.606 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-17T01:51:04.073 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-17T01:51:04.271 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-17T01:51:04.838 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-17T01:51:04.886 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-09-17T01:51:05.004 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2021-09-17T01:51:05.028 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-7730.g2a911e4 8/8 2021-09-17T01:51:05.291 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-7730.g2a911e4 8/8 2021-09-17T01:51:05.292 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-7730.g2a911e4 1/8 2021-09-17T01:51:05.292 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 2/8 2021-09-17T01:51:05.292 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 3/8 2021-09-17T01:51:05.293 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-17T01:51:05.293 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 5/8 2021-09-17T01:51:05.293 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-17T01:51:05.293 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2021-09-17T01:51:05.582 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2021-09-17T01:51:05.583 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:51:05.792 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2021-09-17T01:51:06.093 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-09-17T01:51:06.312 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-17T01:51:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-17T01:51:09.784 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-09-17T01:51:10.011 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-7730.g2a911e4 8/8 2021-09-17T01:51:10.263 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-7730.g2a911e4 8/8 2021-09-17T01:51:10.263 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-7730.g2a911e4 1/8 2021-09-17T01:51:10.264 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 2/8 2021-09-17T01:51:10.264 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 3/8 2021-09-17T01:51:10.264 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-17T01:51:10.264 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 5/8 2021-09-17T01:51:10.265 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-17T01:51:10.265 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2021-09-17T01:51:10.551 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2021-09-17T01:51:10.551 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:51:16.610 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:50:33 AM UTC. 2021-09-17T01:51:19.396 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:51:19.397 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:51:19.397 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr-diskprediction-local-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:51:19.397 INFO:teuthology.orchestra.run.smithi013.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-09-17T01:51:19.397 INFO:teuthology.orchestra.run.smithi013.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-09-17T01:51:19.397 INFO:teuthology.orchestra.run.smithi013.stdout: openblas-0.3.12-1.el8.x86_64 2021-09-17T01:51:19.398 INFO:teuthology.orchestra.run.smithi013.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-09-17T01:51:19.398 INFO:teuthology.orchestra.run.smithi013.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-09-17T01:51:19.398 INFO:teuthology.orchestra.run.smithi013.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-09-17T01:51:19.398 INFO:teuthology.orchestra.run.smithi013.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-09-17T01:51:19.399 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:51:19.399 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:51:19.519 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install ceph-mgr-rook 2021-09-17T01:51:19.869 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:51:20.399 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:51:21.532 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:50:38 AM UTC. 2021-09-17T01:51:24.157 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:51:24.158 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:51:24.158 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:51:24.158 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-09-17T01:51:24.159 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-09-17T01:51:24.159 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.12-1.el8.x86_64 2021-09-17T01:51:24.159 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-09-17T01:51:24.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-09-17T01:51:24.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-09-17T01:51:24.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-09-17T01:51:24.160 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:51:24.160 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:51:24.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-rook 2021-09-17T01:51:24.640 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:51:25.201 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:51:25.531 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-09-17T01:51:30.490 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-17T01:51:31.243 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:51:34.608 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:51:34.609 INFO:teuthology.orchestra.run.smithi013.stdout:======================================================================================================== 2021-09-17T01:51:34.610 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repository Size 2021-09-17T01:51:34.610 INFO:teuthology.orchestra.run.smithi013.stdout:======================================================================================================== 2021-09-17T01:51:34.611 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:51:34.611 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr-rook noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 45 k 2021-09-17T01:51:34.611 INFO:teuthology.orchestra.run.smithi013.stdout:Installing dependencies: 2021-09-17T01:51:34.612 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-09-17T01:51:34.612 INFO:teuthology.orchestra.run.smithi013.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-09-17T01:51:34.612 INFO:teuthology.orchestra.run.smithi013.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-09-17T01:51:34.613 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-09-17T01:51:34.613 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-09-17T01:51:34.613 INFO:teuthology.orchestra.run.smithi013.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-09-17T01:51:34.613 INFO:teuthology.orchestra.run.smithi013.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-09-17T01:51:34.614 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-09-17T01:51:34.614 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-09-17T01:51:34.614 INFO:teuthology.orchestra.run.smithi013.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-17T01:51:34.614 INFO:teuthology.orchestra.run.smithi013.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-09-17T01:51:34.615 INFO:teuthology.orchestra.run.smithi013.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-09-17T01:51:34.615 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:51:34.615 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:51:34.615 INFO:teuthology.orchestra.run.smithi013.stdout:======================================================================================================== 2021-09-17T01:51:34.616 INFO:teuthology.orchestra.run.smithi013.stdout:Install 13 Packages 2021-09-17T01:51:34.616 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:51:34.617 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 1.9 M 2021-09-17T01:51:34.617 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 25 M 2021-09-17T01:51:34.618 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:51:34.687 INFO:teuthology.orchestra.run.smithi013.stdout:(1/13): python3-cachetools-3.1.1-4.el8.noarch.r 448 kB/s | 33 kB 00:00 2021-09-17T01:51:34.697 INFO:teuthology.orchestra.run.smithi013.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 195 kB/s | 16 kB 00:00 2021-09-17T01:51:34.735 INFO:teuthology.orchestra.run.smithi013.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.7 MB/s | 81 kB 00:00 2021-09-17T01:51:34.744 INFO:teuthology.orchestra.run.smithi013.stdout:(4/13): ceph-mgr-rook-17.0.0-7730.g2a911e49.el8 340 kB/s | 45 kB 00:00 2021-09-17T01:51:34.759 INFO:teuthology.orchestra.run.smithi013.stdout:(5/13): python3-rsa-3.4.2-16.el8.noarch.rpm 2.9 MB/s | 73 kB 00:00 2021-09-17T01:51:34.799 INFO:teuthology.orchestra.run.smithi013.stdout:(6/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 682 kB/s | 27 kB 00:00 2021-09-17T01:51:34.826 INFO:teuthology.orchestra.run.smithi013.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 4.0 MB/s | 110 kB 00:00 2021-09-17T01:51:34.845 INFO:teuthology.orchestra.run.smithi013.stdout:(8/13): python3-websocket-client-0.56.0-5.el8.n 613 kB/s | 61 kB 00:00 2021-09-17T01:51:34.856 INFO:teuthology.orchestra.run.smithi013.stdout:(9/13): python3-jsonpointer-1.10-11.el8.noarch. 668 kB/s | 20 kB 00:00 2021-09-17T01:51:34.887 INFO:teuthology.orchestra.run.smithi013.stdout:(10/13): python3-requests-oauthlib-1.0.0-1.el8. 1.4 MB/s | 43 kB 00:00 2021-09-17T01:51:34.897 INFO:teuthology.orchestra.run.smithi013.stdout:(11/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.4 MB/s | 126 kB 00:00 2021-09-17T01:51:34.901 INFO:teuthology.orchestra.run.smithi013.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 5.7 MB/s | 1.2 MB 00:00 2021-09-17T01:51:34.916 INFO:teuthology.orchestra.run.smithi013.stdout:(13/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 5.1 MB/s | 155 kB 00:00 2021-09-17T01:51:34.917 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:51:34.918 INFO:teuthology.orchestra.run.smithi013.stdout:Total 6.3 MB/s | 1.9 MB 00:00 2021-09-17T01:51:34.923 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:51:34.938 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:51:34.939 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:51:35.014 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:51:35.018 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:51:35.631 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:51:35.860 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-09-17T01:51:36.091 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-09-17T01:51:36.106 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:51:36.319 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-09-17T01:51:36.532 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2021-09-17T01:51:36.698 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2021-09-17T01:51:36.875 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2021-09-17T01:51:37.058 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2021-09-17T01:51:37.242 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-09-17T01:51:37.436 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-09-17T01:51:37.660 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-09-17T01:51:38.745 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:51:38.746 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================================== 2021-09-17T01:51:38.746 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-09-17T01:51:38.747 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================================== 2021-09-17T01:51:38.747 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:51:38.747 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 45 k 2021-09-17T01:51:38.747 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-09-17T01:51:38.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-09-17T01:51:38.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-09-17T01:51:38.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-09-17T01:51:38.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-09-17T01:51:38.748 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-09-17T01:51:38.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-09-17T01:51:38.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-09-17T01:51:38.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-09-17T01:51:38.749 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-09-17T01:51:38.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-17T01:51:38.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-09-17T01:51:38.750 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-09-17T01:51:38.750 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:51:38.750 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:51:38.751 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================================== 2021-09-17T01:51:38.751 INFO:teuthology.orchestra.run.smithi045.stdout:Install 13 Packages 2021-09-17T01:51:38.751 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:51:38.752 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 1.9 M 2021-09-17T01:51:38.752 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 25 M 2021-09-17T01:51:38.752 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:51:38.833 INFO:teuthology.orchestra.run.smithi045.stdout:(1/13): python3-cachetools-3.1.1-4.el8.noarch.r 395 kB/s | 33 kB 00:00 2021-09-17T01:51:38.834 INFO:teuthology.orchestra.run.smithi045.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 194 kB/s | 16 kB 00:00 2021-09-17T01:51:38.881 INFO:teuthology.orchestra.run.smithi045.stdout:(3/13): ceph-mgr-rook-17.0.0-7730.g2a911e49.el8 340 kB/s | 45 kB 00:00 2021-09-17T01:51:38.888 INFO:teuthology.orchestra.run.smithi045.stdout:(4/13): python3-google-auth-1.1.1-10.el8.noarch 1.4 MB/s | 81 kB 00:00 2021-09-17T01:51:38.918 INFO:teuthology.orchestra.run.smithi045.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 2.1 MB/s | 61 kB 00:00 2021-09-17T01:51:38.957 INFO:teuthology.orchestra.run.smithi045.stdout:(6/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 677 kB/s | 27 kB 00:00 2021-09-17T01:51:38.975 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-09-17T01:51:38.986 INFO:teuthology.orchestra.run.smithi045.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 3.7 MB/s | 110 kB 00:00 2021-09-17T01:51:38.997 INFO:teuthology.orchestra.run.smithi045.stdout:(8/13): python3-rsa-3.4.2-16.el8.noarch.rpm 632 kB/s | 73 kB 00:00 2021-09-17T01:51:39.024 INFO:teuthology.orchestra.run.smithi045.stdout:(9/13): python3-jsonpointer-1.10-11.el8.noarch. 535 kB/s | 20 kB 00:00 2021-09-17T01:51:39.041 INFO:teuthology.orchestra.run.smithi045.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.8 MB/s | 126 kB 00:00 2021-09-17T01:51:39.061 INFO:teuthology.orchestra.run.smithi045.stdout:(11/13): python3-kubernetes-11.0.0-6.el8.noarch 5.1 MB/s | 1.2 MB 00:00 2021-09-17T01:51:39.068 INFO:teuthology.orchestra.run.smithi045.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 973 kB/s | 43 kB 00:00 2021-09-17T01:51:39.078 INFO:teuthology.orchestra.run.smithi045.stdout:(13/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 4.1 MB/s | 155 kB 00:00 2021-09-17T01:51:39.079 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:51:39.079 INFO:teuthology.orchestra.run.smithi045.stdout:Total 5.8 MB/s | 1.9 MB 00:00 2021-09-17T01:51:39.085 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:51:39.100 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:51:39.101 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:51:39.176 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:51:39.180 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:51:39.284 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-09-17T01:51:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:51:39.949 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-mgr-rook-2:17.0.0-7730.g2a911e49.el8.noarch 13/13 2021-09-17T01:51:39.993 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-09-17T01:51:40.199 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-7730.g2a911e49.el8.noarch 13/13 2021-09-17T01:51:40.200 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-mgr-rook-2:17.0.0-7730.g2a911e49.el8.noarch 1/13 2021-09-17T01:51:40.200 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-09-17T01:51:40.200 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-09-17T01:51:40.201 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-09-17T01:51:40.201 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-09-17T01:51:40.202 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-09-17T01:51:40.202 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-09-17T01:51:40.203 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2021-09-17T01:51:40.203 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/13 2021-09-17T01:51:40.204 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 10/13 2021-09-17T01:51:40.204 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2021-09-17T01:51:40.205 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 12/13 2021-09-17T01:51:40.233 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-09-17T01:51:40.524 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 13/13 2021-09-17T01:51:40.524 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:51:40.529 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-09-17T01:51:40.774 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2021-09-17T01:51:41.066 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2021-09-17T01:51:42.430 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2021-09-17T01:51:42.643 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2021-09-17T01:51:42.867 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-09-17T01:51:43.136 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-09-17T01:51:43.384 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-09-17T01:51:44.616 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-09-17T01:51:44.901 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-09-17T01:51:45.089 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:17.0.0-7730.g2a911e49.el8.noarch 13/13 2021-09-17T01:51:45.329 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-7730.g2a911e49.el8.noarch 13/13 2021-09-17T01:51:45.329 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:17.0.0-7730.g2a911e49.el8.noarch 1/13 2021-09-17T01:51:45.330 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-09-17T01:51:45.330 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-09-17T01:51:45.330 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-09-17T01:51:45.330 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-09-17T01:51:45.331 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-09-17T01:51:45.331 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-09-17T01:51:45.331 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2021-09-17T01:51:45.331 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/13 2021-09-17T01:51:45.331 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 10/13 2021-09-17T01:51:45.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2021-09-17T01:51:45.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 12/13 2021-09-17T01:51:45.661 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 13/13 2021-09-17T01:51:45.661 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:51:51.508 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:51:11 AM UTC. 2021-09-17T01:51:54.126 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:51:54.127 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:51:54.127 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr-rook-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:51:54.127 INFO:teuthology.orchestra.run.smithi013.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-09-17T01:51:54.127 INFO:teuthology.orchestra.run.smithi013.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-09-17T01:51:54.128 INFO:teuthology.orchestra.run.smithi013.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-09-17T01:51:54.128 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-09-17T01:51:54.128 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-09-17T01:51:54.128 INFO:teuthology.orchestra.run.smithi013.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-09-17T01:51:54.128 INFO:teuthology.orchestra.run.smithi013.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-09-17T01:51:54.129 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-09-17T01:51:54.129 INFO:teuthology.orchestra.run.smithi013.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-09-17T01:51:54.129 INFO:teuthology.orchestra.run.smithi013.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-09-17T01:51:54.129 INFO:teuthology.orchestra.run.smithi013.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-09-17T01:51:54.129 INFO:teuthology.orchestra.run.smithi013.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-09-17T01:51:54.130 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:51:54.130 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:51:54.243 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install ceph-mgr-cephadm 2021-09-17T01:51:54.603 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:51:55.157 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 72 kB/s | 2.8 kB 00:00 2021-09-17T01:51:56.838 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:51:16 AM UTC. 2021-09-17T01:51:59.424 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:51:59.424 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:51:59.425 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:51:59.425 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-09-17T01:51:59.425 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-09-17T01:51:59.425 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-09-17T01:51:59.426 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-09-17T01:51:59.426 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-09-17T01:51:59.426 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-09-17T01:51:59.426 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-09-17T01:51:59.427 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-09-17T01:51:59.427 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-09-17T01:51:59.427 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-09-17T01:51:59.427 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-09-17T01:51:59.427 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-09-17T01:51:59.428 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:51:59.428 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:51:59.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-cephadm 2021-09-17T01:52:00.044 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:52:00.567 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:52:00.635 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-09-17T01:52:05.698 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-17T01:52:06.360 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:52:09.065 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:52:09.066 INFO:teuthology.orchestra.run.smithi013.stdout:=============================================================================================== 2021-09-17T01:52:09.066 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repository Size 2021-09-17T01:52:09.067 INFO:teuthology.orchestra.run.smithi013.stdout:=============================================================================================== 2021-09-17T01:52:09.067 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:52:09.067 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr-cephadm noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 91 k 2021-09-17T01:52:09.067 INFO:teuthology.orchestra.run.smithi013.stdout:Installing dependencies: 2021-09-17T01:52:09.068 INFO:teuthology.orchestra.run.smithi013.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2021-09-17T01:52:09.068 INFO:teuthology.orchestra.run.smithi013.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-09-17T01:52:09.068 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-09-17T01:52:09.068 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:52:09.068 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:52:09.069 INFO:teuthology.orchestra.run.smithi013.stdout:=============================================================================================== 2021-09-17T01:52:09.069 INFO:teuthology.orchestra.run.smithi013.stdout:Install 4 Packages 2021-09-17T01:52:09.069 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:52:09.070 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 5.9 M 2021-09-17T01:52:09.070 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 25 M 2021-09-17T01:52:09.070 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:52:09.114 INFO:teuthology.orchestra.run.smithi013.stdout:(1/4): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 11 MB/s | 538 kB 00:00 2021-09-17T01:52:09.172 INFO:teuthology.orchestra.run.smithi013.stdout:(2/4): python3-babel-2.5.1-5.el8.noarch.rpm 83 MB/s | 4.8 MB 00:00 2021-09-17T01:52:09.228 INFO:teuthology.orchestra.run.smithi013.stdout:(3/4): ceph-mgr-cephadm-17.0.0-7730.g2a911e49.e 570 kB/s | 91 kB 00:00 2021-09-17T01:52:09.251 INFO:teuthology.orchestra.run.smithi013.stdout:(4/4): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.5 MB/s | 476 kB 00:00 2021-09-17T01:52:09.252 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:52:09.253 INFO:teuthology.orchestra.run.smithi013.stdout:Total 32 MB/s | 5.9 MB 00:00 2021-09-17T01:52:09.256 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:52:09.265 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:52:09.266 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:52:09.345 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:52:09.349 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:52:10.353 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:52:11.037 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/4 2021-09-17T01:52:11.308 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:52:11.402 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/4 2021-09-17T01:52:11.636 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 3/4 2021-09-17T01:52:11.803 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 4/4 2021-09-17T01:52:12.035 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 4/4 2021-09-17T01:52:12.035 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 1/4 2021-09-17T01:52:12.036 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/4 2021-09-17T01:52:12.036 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-17T01:52:12.297 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/4 2021-09-17T01:52:12.298 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:52:14.085 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:52:14.085 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================================== 2021-09-17T01:52:14.086 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-09-17T01:52:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================================== 2021-09-17T01:52:14.086 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:52:14.086 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:17.0.0-7730.g2a911e49.el8 ceph-noarch 91 k 2021-09-17T01:52:14.087 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-09-17T01:52:14.087 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2021-09-17T01:52:14.087 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-09-17T01:52:14.088 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-09-17T01:52:14.088 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:52:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:52:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================================== 2021-09-17T01:52:14.089 INFO:teuthology.orchestra.run.smithi045.stdout:Install 4 Packages 2021-09-17T01:52:14.089 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:52:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 5.9 M 2021-09-17T01:52:14.090 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 25 M 2021-09-17T01:52:14.091 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:52:14.130 INFO:teuthology.orchestra.run.smithi045.stdout:(1/4): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 13 MB/s | 538 kB 00:00 2021-09-17T01:52:14.190 INFO:teuthology.orchestra.run.smithi045.stdout:(2/4): python3-babel-2.5.1-5.el8.noarch.rpm 80 MB/s | 4.8 MB 00:00 2021-09-17T01:52:14.345 INFO:teuthology.orchestra.run.smithi045.stdout:(3/4): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.8 MB/s | 476 kB 00:00 2021-09-17T01:52:14.393 INFO:teuthology.orchestra.run.smithi045.stdout:(4/4): ceph-mgr-cephadm-17.0.0-7730.g2a911e49.e 299 kB/s | 91 kB 00:00 2021-09-17T01:52:14.394 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:52:14.395 INFO:teuthology.orchestra.run.smithi045.stdout:Total 19 MB/s | 5.9 MB 00:00 2021-09-17T01:52:14.398 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:52:14.407 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:52:14.408 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:52:14.487 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:52:14.491 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:52:15.508 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:52:15.922 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/4 2021-09-17T01:52:16.316 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/4 2021-09-17T01:52:16.603 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 3/4 2021-09-17T01:52:16.851 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 4/4 2021-09-17T01:52:17.120 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 4/4 2021-09-17T01:52:17.120 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 1/4 2021-09-17T01:52:17.121 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/4 2021-09-17T01:52:17.121 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-17T01:52:17.392 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/4 2021-09-17T01:52:17.392 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:52:23.893 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:51:46 AM UTC. 2021-09-17T01:52:26.608 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:52:26.609 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:52:26.609 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-mgr-cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:52:26.609 INFO:teuthology.orchestra.run.smithi013.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2021-09-17T01:52:26.610 INFO:teuthology.orchestra.run.smithi013.stdout: python3-babel-2.5.1-5.el8.noarch 2021-09-17T01:52:26.610 INFO:teuthology.orchestra.run.smithi013.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-09-17T01:52:26.610 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:52:26.610 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:52:26.730 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install ceph-fuse 2021-09-17T01:52:27.089 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:52:27.623 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-17T01:52:28.491 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:51:51 AM UTC. 2021-09-17T01:52:31.177 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:52:31.177 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:52:31.178 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:17.0.0-7730.g2a911e49.el8.noarch 2021-09-17T01:52:31.178 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2021-09-17T01:52:31.178 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.5.1-5.el8.noarch 2021-09-17T01:52:31.178 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-09-17T01:52:31.179 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:52:31.179 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:52:31.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-fuse 2021-09-17T01:52:31.659 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:52:32.204 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-17T01:52:32.862 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-09-17T01:52:37.321 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-17T01:52:38.529 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:52:41.197 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:52:41.198 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:52:41.198 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repository Size 2021-09-17T01:52:41.198 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:52:41.199 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:52:41.199 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-fuse 2021-09-17T01:52:41.199 INFO:teuthology.orchestra.run.smithi013.stdout: x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 867 k 2021-09-17T01:52:41.199 INFO:teuthology.orchestra.run.smithi013.stdout:Installing dependencies: 2021-09-17T01:52:41.200 INFO:teuthology.orchestra.run.smithi013.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-09-17T01:52:41.200 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:52:41.200 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:52:41.200 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:52:41.201 INFO:teuthology.orchestra.run.smithi013.stdout:Install 2 Packages 2021-09-17T01:52:41.201 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:52:41.201 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 950 k 2021-09-17T01:52:41.201 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 3.1 M 2021-09-17T01:52:41.202 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:52:41.243 INFO:teuthology.orchestra.run.smithi013.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.8 MB/s | 83 kB 00:00 2021-09-17T01:52:41.439 INFO:teuthology.orchestra.run.smithi013.stdout:(2/2): ceph-fuse-17.0.0-7730.g2a911e49.el8.x86_ 3.5 MB/s | 867 kB 00:00 2021-09-17T01:52:41.439 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:52:41.440 INFO:teuthology.orchestra.run.smithi013.stdout:Total 3.8 MB/s | 950 kB 00:00 2021-09-17T01:52:41.441 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:52:41.464 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:52:41.465 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:52:41.545 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:52:41.550 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:52:41.713 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:52:42.248 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-09-17T01:52:42.424 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : ceph-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 2/2 2021-09-17T01:52:42.802 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: ceph-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 2/2 2021-09-17T01:52:42.803 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : ceph-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 1/2 2021-09-17T01:52:42.991 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:52:43.069 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-09-17T01:52:43.069 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:52:45.631 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:52:45.632 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:52:45.632 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2021-09-17T01:52:45.632 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:52:45.633 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:52:45.633 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse 2021-09-17T01:52:45.633 INFO:teuthology.orchestra.run.smithi045.stdout: x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 867 k 2021-09-17T01:52:45.633 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2021-09-17T01:52:45.633 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-09-17T01:52:45.634 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:52:45.634 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:52:45.634 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:52:45.634 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2021-09-17T01:52:45.635 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:52:45.635 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 950 k 2021-09-17T01:52:45.635 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 3.1 M 2021-09-17T01:52:45.635 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:52:45.678 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.8 MB/s | 83 kB 00:00 2021-09-17T01:52:45.876 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-fuse-17.0.0-7730.g2a911e49.el8.x86_ 3.5 MB/s | 867 kB 00:00 2021-09-17T01:52:45.877 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:52:45.877 INFO:teuthology.orchestra.run.smithi045.stdout:Total 3.8 MB/s | 950 kB 00:00 2021-09-17T01:52:45.879 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:52:45.902 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:52:45.902 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:52:45.980 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:52:45.984 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:52:46.138 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:52:46.411 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-09-17T01:52:46.562 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 2/2 2021-09-17T01:52:46.918 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 2/2 2021-09-17T01:52:46.918 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 1/2 2021-09-17T01:52:47.176 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-09-17T01:52:47.176 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:52:54.249 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:52:18 AM UTC. 2021-09-17T01:52:56.822 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:52:56.822 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:52:56.823 INFO:teuthology.orchestra.run.smithi013.stdout: ceph-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-09-17T01:52:56.823 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:52:56.823 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:52:56.952 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install librados-devel 2021-09-17T01:52:57.314 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:52:57.855 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-17T01:52:58.224 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:52:22 AM UTC. 2021-09-17T01:53:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:53:00.876 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:53:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-09-17T01:53:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:53:00.877 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:53:01.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados-devel 2021-09-17T01:53:01.405 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:53:01.953 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 72 kB/s | 2.8 kB 00:00 2021-09-17T01:53:03.109 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-09-17T01:53:07.117 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-09-17T01:53:08.841 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:53:11.459 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:53:11.460 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:53:11.461 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repo Size 2021-09-17T01:53:11.461 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:53:11.461 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:53:11.461 INFO:teuthology.orchestra.run.smithi013.stdout: librados-devel x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 125 k 2021-09-17T01:53:11.461 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:53:11.462 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:53:11.462 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:53:11.462 INFO:teuthology.orchestra.run.smithi013.stdout:Install 1 Package 2021-09-17T01:53:11.462 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:53:11.463 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 125 k 2021-09-17T01:53:11.463 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 432 k 2021-09-17T01:53:11.463 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:53:11.631 INFO:teuthology.orchestra.run.smithi013.stdout:librados-devel-17.0.0-7730.g2a911e49.el8.x86_64 734 kB/s | 125 kB 00:00 2021-09-17T01:53:11.632 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:53:11.632 INFO:teuthology.orchestra.run.smithi013.stdout:Total 728 kB/s | 125 kB 00:00 2021-09-17T01:53:11.632 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:53:11.639 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:53:11.639 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:53:11.700 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:53:11.705 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:53:11.822 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:53:11.976 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : librados-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:12.211 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: librados-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:12.442 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : librados-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:12.443 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:53:12.768 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:53:15.393 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:53:15.394 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:53:15.395 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2021-09-17T01:53:15.395 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:53:15.395 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:53:15.395 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 125 k 2021-09-17T01:53:15.396 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:53:15.396 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:53:15.396 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:53:15.396 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-09-17T01:53:15.396 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:53:15.397 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 125 k 2021-09-17T01:53:15.397 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 432 k 2021-09-17T01:53:15.397 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:53:15.549 INFO:teuthology.orchestra.run.smithi045.stdout:librados-devel-17.0.0-7730.g2a911e49.el8.x86_64 810 kB/s | 125 kB 00:00 2021-09-17T01:53:15.550 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:53:15.550 INFO:teuthology.orchestra.run.smithi045.stdout:Total 802 kB/s | 125 kB 00:00 2021-09-17T01:53:15.551 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:53:15.558 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:53:15.559 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:53:15.619 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:53:15.623 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:53:15.924 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:53:16.082 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:16.304 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:16.551 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:16.552 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:53:23.426 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:52:48 AM UTC. 2021-09-17T01:53:26.100 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:53:26.101 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:53:26.101 INFO:teuthology.orchestra.run.smithi013.stdout: librados-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:53:26.101 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:53:26.101 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:53:26.223 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install libcephfs2 2021-09-17T01:53:26.586 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:53:27.126 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-17T01:53:27.495 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:52:52 AM UTC. 2021-09-17T01:53:30.179 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:53:30.179 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:53:30.180 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:53:30.180 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:53:30.180 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:53:30.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs2 2021-09-17T01:53:30.665 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:53:31.193 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-17T01:53:32.393 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-17T01:53:36.330 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-09-17T01:53:38.076 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:53:40.270 INFO:teuthology.orchestra.run.smithi013.stdout:Package libcephfs2-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:53:40.722 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:53:40.723 INFO:teuthology.orchestra.run.smithi013.stdout:Nothing to do. 2021-09-17T01:53:40.724 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:53:40.829 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install libcephfs-devel 2021-09-17T01:53:41.304 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:53:41.861 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-09-17T01:53:42.053 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-17T01:53:42.198 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:53:44.020 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:53:44.021 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:53:44.022 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repo Size 2021-09-17T01:53:44.022 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:53:44.022 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:53:44.022 INFO:teuthology.orchestra.run.smithi013.stdout: libcephfs-devel x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 24 k 2021-09-17T01:53:44.023 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:53:44.023 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:53:44.023 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:53:44.023 INFO:teuthology.orchestra.run.smithi013.stdout:Install 1 Package 2021-09-17T01:53:44.024 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:53:44.024 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 24 k 2021-09-17T01:53:44.024 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 102 k 2021-09-17T01:53:44.024 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:53:44.134 INFO:teuthology.orchestra.run.smithi013.stdout:libcephfs-devel-17.0.0-7730.g2a911e49.el8.x86_6 214 kB/s | 24 kB 00:00 2021-09-17T01:53:44.135 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:53:44.135 INFO:teuthology.orchestra.run.smithi013.stdout:Total 212 kB/s | 24 kB 00:00 2021-09-17T01:53:44.135 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:53:44.139 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:53:44.139 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:53:44.141 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:53:44.145 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:53:44.199 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:53:44.347 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : libcephfs-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:44.530 INFO:teuthology.orchestra.run.smithi045.stdout:Package libcephfs2-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:53:44.562 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:44.806 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : libcephfs-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:44.806 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:53:44.979 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:53:44.980 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-09-17T01:53:44.980 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:53:45.092 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs-devel 2021-09-17T01:53:45.449 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:53:46.000 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:53:46.192 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-17T01:53:48.193 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:53:48.194 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:53:48.194 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2021-09-17T01:53:48.194 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:53:48.194 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:53:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 24 k 2021-09-17T01:53:48.195 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:53:48.195 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:53:48.195 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:53:48.196 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-09-17T01:53:48.196 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:53:48.196 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 24 k 2021-09-17T01:53:48.196 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 102 k 2021-09-17T01:53:48.196 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:53:48.322 INFO:teuthology.orchestra.run.smithi045.stdout:libcephfs-devel-17.0.0-7730.g2a911e49.el8.x86_6 188 kB/s | 24 kB 00:00 2021-09-17T01:53:48.323 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:53:48.323 INFO:teuthology.orchestra.run.smithi045.stdout:Total 185 kB/s | 24 kB 00:00 2021-09-17T01:53:48.324 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:53:48.327 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:53:48.327 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:53:48.329 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:53:48.333 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:53:48.391 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:53:48.552 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:48.795 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:49.040 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:53:49.040 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:53:55.827 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:53:17 AM UTC. 2021-09-17T01:53:58.469 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:53:58.469 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:53:58.469 INFO:teuthology.orchestra.run.smithi013.stdout: libcephfs-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:53:58.470 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:53:58.470 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:53:58.588 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install librados2 2021-09-17T01:53:58.945 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:53:59.485 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-17T01:54:00.087 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:53:22 AM UTC. 2021-09-17T01:54:02.728 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:54:02.729 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:54:02.729 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:54:02.729 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:54:02.729 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:54:02.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados2 2021-09-17T01:54:03.215 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:03.825 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:54:04.569 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-09-17T01:54:08.961 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-09-17T01:54:10.309 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:54:12.482 INFO:teuthology.orchestra.run.smithi013.stdout:Package librados2-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:12.928 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:54:12.929 INFO:teuthology.orchestra.run.smithi013.stdout:Nothing to do. 2021-09-17T01:54:12.930 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:54:13.026 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install librbd1 2021-09-17T01:54:13.384 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:13.924 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-09-17T01:54:14.115 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-09-17T01:54:14.533 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:54:15.592 INFO:teuthology.orchestra.run.smithi013.stdout:Package librbd1-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:16.045 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:54:16.046 INFO:teuthology.orchestra.run.smithi013.stdout:Nothing to do. 2021-09-17T01:54:16.046 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:54:16.150 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install python3-rados 2021-09-17T01:54:16.508 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:16.741 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:17.035 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:54:17.188 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:54:17.189 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-09-17T01:54:17.189 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:54:17.223 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-17T01:54:17.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librbd1 2021-09-17T01:54:17.647 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:18.189 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:54:18.380 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-17T01:54:18.639 INFO:teuthology.orchestra.run.smithi013.stdout:Package python3-rados-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:19.088 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:54:19.089 INFO:teuthology.orchestra.run.smithi013.stdout:Nothing to do. 2021-09-17T01:54:19.089 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:54:19.180 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install python3-rgw 2021-09-17T01:54:19.540 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:19.899 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:20.074 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 72 kB/s | 2.8 kB 00:00 2021-09-17T01:54:20.266 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-17T01:54:20.352 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:54:20.353 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-09-17T01:54:20.354 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:54:20.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rados 2021-09-17T01:54:20.812 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:21.357 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:54:21.546 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-17T01:54:21.689 INFO:teuthology.orchestra.run.smithi013.stdout:Package python3-rgw-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:22.138 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:54:22.139 INFO:teuthology.orchestra.run.smithi013.stdout:Nothing to do. 2021-09-17T01:54:22.139 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:54:22.240 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install python3-cephfs 2021-09-17T01:54:22.596 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:23.065 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rados-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:23.170 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 72 kB/s | 2.8 kB 00:00 2021-09-17T01:54:23.366 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 kB/s | 2.4 kB 00:00 2021-09-17T01:54:23.511 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:54:23.513 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-09-17T01:54:23.513 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:54:23.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rgw 2021-09-17T01:54:24.079 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:24.612 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 77 kB/s | 2.8 kB 00:00 2021-09-17T01:54:24.788 INFO:teuthology.orchestra.run.smithi013.stdout:Package python3-cephfs-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:24.804 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-17T01:54:25.232 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:54:25.233 INFO:teuthology.orchestra.run.smithi013.stdout:Nothing to do. 2021-09-17T01:54:25.233 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:54:25.323 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install python3-rbd 2021-09-17T01:54:25.680 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:26.213 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-17T01:54:26.365 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rgw-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:26.405 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-17T01:54:26.821 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:54:26.822 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-09-17T01:54:26.822 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:54:26.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-cephfs 2021-09-17T01:54:27.279 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:27.821 INFO:teuthology.orchestra.run.smithi013.stdout:Package python3-rbd-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:27.825 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:54:28.013 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-17T01:54:28.277 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:54:28.278 INFO:teuthology.orchestra.run.smithi013.stdout:Nothing to do. 2021-09-17T01:54:28.278 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:54:28.384 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install rbd-fuse 2021-09-17T01:54:28.741 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:29.265 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 79 kB/s | 2.8 kB 00:00 2021-09-17T01:54:29.457 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-17T01:54:29.497 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-cephfs-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:29.948 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:54:29.950 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-09-17T01:54:29.950 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:54:30.060 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rbd 2021-09-17T01:54:30.419 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:30.943 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-09-17T01:54:31.140 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-09-17T01:54:31.428 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:54:31.429 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:54:31.429 INFO:teuthology.orchestra.run.smithi013.stdout: Package Architecture Version Repository Size 2021-09-17T01:54:31.429 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:54:31.430 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:54:31.430 INFO:teuthology.orchestra.run.smithi013.stdout: rbd-fuse x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 94 k 2021-09-17T01:54:31.430 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:54:31.431 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:54:31.431 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:54:31.431 INFO:teuthology.orchestra.run.smithi013.stdout:Install 1 Package 2021-09-17T01:54:31.431 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:54:31.432 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 94 k 2021-09-17T01:54:31.432 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 281 k 2021-09-17T01:54:31.432 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:54:31.586 INFO:teuthology.orchestra.run.smithi013.stdout:rbd-fuse-17.0.0-7730.g2a911e49.el8.x86_64.rpm 606 kB/s | 94 kB 00:00 2021-09-17T01:54:31.587 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:54:31.587 INFO:teuthology.orchestra.run.smithi013.stdout:Total 600 kB/s | 94 kB 00:00 2021-09-17T01:54:31.587 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:54:31.598 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:54:31.598 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:54:31.657 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:54:31.661 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:54:31.775 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:54:31.942 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : rbd-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:54:32.178 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: rbd-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:54:32.450 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : rbd-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:54:32.451 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:54:32.631 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rbd-2:17.0.0-7730.g2a911e49.el8.x86_64 is already installed. 2021-09-17T01:54:33.074 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:54:33.075 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2021-09-17T01:54:33.075 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:54:33.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-fuse 2021-09-17T01:54:33.537 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:34.061 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:54:34.254 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-09-17T01:54:36.270 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:54:36.271 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:54:36.271 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2021-09-17T01:54:36.272 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:54:36.272 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:54:36.272 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 94 k 2021-09-17T01:54:36.272 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:54:36.273 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:54:36.273 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:54:36.273 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-09-17T01:54:36.273 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:54:36.274 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 94 k 2021-09-17T01:54:36.274 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 281 k 2021-09-17T01:54:36.274 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:54:36.423 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-fuse-17.0.0-7730.g2a911e49.el8.x86_64.rpm 615 kB/s | 94 kB 00:00 2021-09-17T01:54:36.425 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:54:36.425 INFO:teuthology.orchestra.run.smithi045.stdout:Total 609 kB/s | 94 kB 00:00 2021-09-17T01:54:36.425 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:54:36.438 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:54:36.438 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:54:36.496 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:54:36.500 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:54:36.614 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:54:36.812 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:54:37.029 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:54:37.275 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:54:37.276 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:54:43.485 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:53:50 AM UTC. 2021-09-17T01:54:46.143 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:54:46.143 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:54:46.143 INFO:teuthology.orchestra.run.smithi013.stdout: rbd-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:54:46.144 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:54:46.144 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:54:46.267 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install rbd-mirror 2021-09-17T01:54:46.635 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:47.212 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:54:48.195 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:53:54 AM UTC. 2021-09-17T01:54:50.775 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:54:50.775 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:54:50.776 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:54:50.776 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:54:50.776 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:54:50.917 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-mirror 2021-09-17T01:54:51.286 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:54:51.821 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-17T01:54:52.401 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-17T01:54:56.933 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-17T01:54:58.158 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:55:00.796 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:55:00.797 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:55:00.798 INFO:teuthology.orchestra.run.smithi013.stdout: Package Arch Version Repo Size 2021-09-17T01:55:00.798 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:55:00.798 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:55:00.798 INFO:teuthology.orchestra.run.smithi013.stdout: rbd-mirror x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 3.8 M 2021-09-17T01:55:00.799 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:00.799 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:55:00.799 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:55:00.799 INFO:teuthology.orchestra.run.smithi013.stdout:Install 1 Package 2021-09-17T01:55:00.799 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:00.800 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 3.8 M 2021-09-17T01:55:00.800 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 19 M 2021-09-17T01:55:00.800 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:55:01.264 INFO:teuthology.orchestra.run.smithi013.stdout:rbd-mirror-17.0.0-7730.g2a911e49.el8.x86_64.rpm 8.1 MB/s | 3.8 MB 00:00 2021-09-17T01:55:01.265 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:55:01.266 INFO:teuthology.orchestra.run.smithi013.stdout:Total 8.0 MB/s | 3.8 MB 00:00 2021-09-17T01:55:01.267 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:55:01.281 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:55:01.281 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:55:01.355 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:55:01.359 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:55:01.874 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:55:02.073 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : rbd-mirror-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:02.456 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: rbd-mirror-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:02.538 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:55:02.723 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : rbd-mirror-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:02.724 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:55:05.216 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:55:05.217 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:55:05.217 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2021-09-17T01:55:05.217 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:55:05.218 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:55:05.218 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 3.8 M 2021-09-17T01:55:05.218 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:05.218 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:55:05.219 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:55:05.219 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-09-17T01:55:05.219 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:05.219 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.8 M 2021-09-17T01:55:05.219 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 19 M 2021-09-17T01:55:05.220 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:55:05.737 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-mirror-17.0.0-7730.g2a911e49.el8.x86_64.rpm 7.2 MB/s | 3.8 MB 00:00 2021-09-17T01:55:05.738 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:55:05.738 INFO:teuthology.orchestra.run.smithi045.stdout:Total 7.2 MB/s | 3.8 MB 00:00 2021-09-17T01:55:05.739 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:55:05.753 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:55:05.754 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:55:05.826 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:55:05.830 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:55:06.336 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:55:06.551 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:06.929 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:07.186 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:07.186 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:55:13.937 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:54:37 AM UTC. 2021-09-17T01:55:16.527 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:16.527 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:55:16.527 INFO:teuthology.orchestra.run.smithi013.stdout: rbd-mirror-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:55:16.528 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:16.528 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:55:16.652 DEBUG:teuthology.orchestra.run.smithi013:> sudo yum -y install rbd-nbd 2021-09-17T01:55:17.013 INFO:teuthology.orchestra.run.smithi013.stdout:Updating Subscription Management repositories. 2021-09-17T01:55:17.584 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-17T01:55:18.326 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:54:42 AM UTC. 2021-09-17T01:55:20.972 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:20.972 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:55:20.973 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:55:20.973 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:20.973 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:55:21.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-nbd 2021-09-17T01:55:21.492 INFO:teuthology.orchestra.run.smithi045.stdout:Updating Subscription Management repositories. 2021-09-17T01:55:22.039 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-09-17T01:55:22.728 INFO:teuthology.orchestra.run.smithi013.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-17T01:55:27.447 INFO:teuthology.orchestra.run.smithi045.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-17T01:55:28.308 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:55:31.035 INFO:teuthology.orchestra.run.smithi013.stdout:Dependencies resolved. 2021-09-17T01:55:31.036 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:55:31.037 INFO:teuthology.orchestra.run.smithi013.stdout: Package Architecture Version Repository Size 2021-09-17T01:55:31.037 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:55:31.037 INFO:teuthology.orchestra.run.smithi013.stdout:Installing: 2021-09-17T01:55:31.037 INFO:teuthology.orchestra.run.smithi013.stdout: rbd-nbd x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 173 k 2021-09-17T01:55:31.038 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:31.038 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction Summary 2021-09-17T01:55:31.038 INFO:teuthology.orchestra.run.smithi013.stdout:================================================================================ 2021-09-17T01:55:31.038 INFO:teuthology.orchestra.run.smithi013.stdout:Install 1 Package 2021-09-17T01:55:31.039 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:31.039 INFO:teuthology.orchestra.run.smithi013.stdout:Total download size: 173 k 2021-09-17T01:55:31.039 INFO:teuthology.orchestra.run.smithi013.stdout:Installed size: 563 k 2021-09-17T01:55:31.039 INFO:teuthology.orchestra.run.smithi013.stdout:Downloading Packages: 2021-09-17T01:55:31.207 INFO:teuthology.orchestra.run.smithi013.stdout:rbd-nbd-17.0.0-7730.g2a911e49.el8.x86_64.rpm 1.0 MB/s | 173 kB 00:00 2021-09-17T01:55:31.208 INFO:teuthology.orchestra.run.smithi013.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:55:31.208 INFO:teuthology.orchestra.run.smithi013.stdout:Total 1.0 MB/s | 173 kB 00:00 2021-09-17T01:55:31.208 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction check 2021-09-17T01:55:31.222 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction check succeeded. 2021-09-17T01:55:31.222 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction test 2021-09-17T01:55:31.282 INFO:teuthology.orchestra.run.smithi013.stdout:Transaction test succeeded. 2021-09-17T01:55:31.286 INFO:teuthology.orchestra.run.smithi013.stdout:Running transaction 2021-09-17T01:55:31.407 INFO:teuthology.orchestra.run.smithi013.stdout: Preparing : 1/1 2021-09-17T01:55:31.592 INFO:teuthology.orchestra.run.smithi013.stdout: Installing : rbd-nbd-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:31.809 INFO:teuthology.orchestra.run.smithi013.stdout: Running scriptlet: rbd-nbd-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:32.049 INFO:teuthology.orchestra.run.smithi013.stdout: Verifying : rbd-nbd-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:32.050 INFO:teuthology.orchestra.run.smithi013.stdout:Installed products updated. 2021-09-17T01:55:33.035 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:46:34 AM UTC. 2021-09-17T01:55:35.713 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2021-09-17T01:55:35.714 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:55:35.714 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2021-09-17T01:55:35.714 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:55:35.714 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2021-09-17T01:55:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:17.0.0-7730.g2a911e49.el8 ceph 173 k 2021-09-17T01:55:35.715 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:35.715 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2021-09-17T01:55:35.715 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2021-09-17T01:55:35.716 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2021-09-17T01:55:35.716 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:35.716 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 173 k 2021-09-17T01:55:35.716 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 563 k 2021-09-17T01:55:35.716 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2021-09-17T01:55:35.886 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-nbd-17.0.0-7730.g2a911e49.el8.x86_64.rpm 1.0 MB/s | 173 kB 00:00 2021-09-17T01:55:35.887 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2021-09-17T01:55:35.887 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.0 MB/s | 173 kB 00:00 2021-09-17T01:55:35.888 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2021-09-17T01:55:35.901 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2021-09-17T01:55:35.902 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2021-09-17T01:55:35.960 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2021-09-17T01:55:35.965 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2021-09-17T01:55:36.088 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2021-09-17T01:55:36.394 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:36.627 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-nbd-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:36.924 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:17.0.0-7730.g2a911e49.el8.x86_64 1/1 2021-09-17T01:55:36.925 INFO:teuthology.orchestra.run.smithi045.stdout:Installed products updated. 2021-09-17T01:55:43.528 INFO:teuthology.orchestra.run.smithi013.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:55:08 AM UTC. 2021-09-17T01:55:46.196 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:46.196 INFO:teuthology.orchestra.run.smithi013.stdout:Installed: 2021-09-17T01:55:46.196 INFO:teuthology.orchestra.run.smithi013.stdout: rbd-nbd-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:55:46.197 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:46.197 INFO:teuthology.orchestra.run.smithi013.stdout:Complete! 2021-09-17T01:55:46.313 DEBUG:teuthology.parallel:result is None 2021-09-17T01:55:47.955 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:05 ago on Fri 17 Sep 2021 01:55:12 AM UTC. 2021-09-17T01:55:50.560 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:50.560 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2021-09-17T01:55:50.560 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:17.0.0-7730.g2a911e49.el8.x86_64 2021-09-17T01:55:50.561 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:50.561 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2021-09-17T01:55:50.686 DEBUG:teuthology.parallel:result is None 2021-09-17T01:55:50.687 DEBUG:teuthology.orchestra.run.smithi013:> sudo lsb_release -is 2021-09-17T01:55:50.735 INFO:teuthology.orchestra.run.smithi013.stdout:RedHatEnterprise 2021-09-17T01:55:50.737 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-17T01:55:50.738 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2a911e49300877ce783c2becf9f484a85fa1c049 2021-09-17T01:55:50.912 DEBUG:teuthology.orchestra.run.smithi013:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-09-17T01:55:50.936 INFO:teuthology.orchestra.run.smithi013.stdout:17.0.0-7730.g2a911e49.el8 2021-09-17T01:55:50.936 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7730.g2a911e49.el8 2021-09-17T01:55:50.937 INFO:teuthology.task.install:The correct ceph version 17.0.0-7730.g2a911e49 is installed. 2021-09-17T01:55:50.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo lsb_release -is 2021-09-17T01:55:50.981 INFO:teuthology.orchestra.run.smithi045.stdout:RedHatEnterprise 2021-09-17T01:55:50.982 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-17T01:55:50.982 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2a911e49300877ce783c2becf9f484a85fa1c049 2021-09-17T01:55:51.130 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-09-17T01:55:51.150 INFO:teuthology.orchestra.run.smithi045.stdout:17.0.0-7730.g2a911e49.el8 2021-09-17T01:55:51.151 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7730.g2a911e49.el8 2021-09-17T01:55:51.151 INFO:teuthology.task.install:The correct ceph version 17.0.0-7730.g2a911e49 is installed. 2021-09-17T01:55:51.152 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-09-17T01:55:51.152 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:55:51.153 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-17T01:55:51.182 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:55:51.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-17T01:55:51.216 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-09-17T01:55:51.217 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:55:51.217 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd of=/usr/bin/daemon-helper 2021-09-17T01:55:51.252 DEBUG:teuthology.orchestra.run.smithi013:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-17T01:55:51.322 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:55:51.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2021-09-17T01:55:51.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-17T01:55:51.413 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-09-17T01:55:51.414 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:55:51.414 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-17T01:55:51.442 DEBUG:teuthology.orchestra.run.smithi013:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-17T01:55:51.511 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:55:51.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-17T01:55:51.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-17T01:55:51.601 INFO:teuthology.run_tasks:Running task ceph... 2021-09-17T01:55:51.676 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-09-17T01:55:51.677 DEBUG:teuthology.orchestra.run.smithi013:> sudo chmod 777 /var/log/ceph 2021-09-17T01:55:51.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /var/log/ceph 2021-09-17T01:55:51.707 INFO:tasks.ceph:Disabling ceph logrotate... 2021-09-17T01:55:51.708 DEBUG:teuthology.orchestra.run.smithi013:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-17T01:55:51.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-17T01:55:51.777 INFO:tasks.ceph:Creating extra log directories... 2021-09-17T01:55:51.778 DEBUG:teuthology.orchestra.run.smithi013:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-17T01:55:51.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-17T01:55:51.863 INFO:tasks.ceph:Creating ceph cluster c1... 2021-09-17T01:55:51.863 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-17T01:55:51.864 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/ivancich-2021-09-16_17:11:22-rgw-wip-eric-testing-2-distro-basic-smithi/6392884', 'branch': 'wip-eric-testing-2', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': 'ivancich@redhat.com', 'first_in_suite': False, 'job_id': '6392884', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'ivancich-2021-09-16_17:11:22-rgw-wip-eric-testing-2-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-eric-testing-2'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-eric-testing-2', 'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049'}}, 'owner': 'scheduled_ivancich@teuthology', 'priority': 83, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-eric-testing-2', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_2a911e49300877ce783c2becf9f484a85fa1c049/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '2a911e49300877ce783c2becf9f484a85fa1c049', 'targets': {'smithi013.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQUSlRIEp7IBLYORccR8lUdTmBJ7wmpIUUDFB3BhcT3ksU2F+8QkMCOspCnZF1sYWLW7px9JhbwBIFpTNh82FaIYm+10eqZjBj0nwoTruKjA2DxTEat/JJX2bMF71TbOrjDQwZccmz8TZGYhIlK2tHMoOPB8rGEcM3AHllSFf64/M7lUiZlIZOOEChRvepzkPTfdyHE7lBCH+PJ9pCabnK17OjVgmBGEkUWbbprw6HadBeQ1CrgVzwjabqyAd/8DCvTY2OxE/k+sVVZe4U6kEdBcRe9erCVlQxjoXlPchpL+k0hA2+rk8LHkurz/mdT9av4QVYxi8YcTKIfjYbwhgq5eu964h+rl3y4oZ/NIk+X8MOzsHViYYfHA5WXCRDcZxqUojCCL5+VntRfbe/cwNK9UGdV4LJ3F9dNSw9tYEAvtzE7Mh7xzHOPntmtg/pL43RboxM/YqQEp5wAGNFLPBjhhfsx+54JpWVZBYw5hjU9m36xlETxeS60ZMpA9ZpiuE=', 'smithi045.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCulDN10/Vke3BN23zcEbwKxvhTR910c5AgG82/Ohf4WiNPYA98666PzF0PZudRGkkX89oUG+zPhqFxQc+j3oGHtRyORPLZzJPG//ZKLtwr1WONI012azN1JgotupRG+CwfHM7sThE09XdXiYLDYsIj9R5NDPT8ATTKxRAMVpMUuPx0MjyS9R7c4mVQ+ZKEDq8hRS1hl5svH+iBtEooVtLGZljHsj8HvHDjdtrrTxbSkzhZxHlnD6pEm2SngtbAglAiMO5NuYltRDf8x/k6V3eBODOz4rJUdDtsif9nCjU2HUEcdnNTyZlRJizeshvySk67OoZ5fc2eezTmluQ6QIcv+B0dLUG3Eor1oqHCdtxzBR/PJMZo9tLppdtH9AHitodCgOkdMJyd/zzLBF9iDLBsrXPQ20wVwb5rqnqzRZIpz28zreE9wrS07Ny5FfkexvGofoqbbDpjm8sshrtwv/jfcPcTYd0NUhL49SuSyhOWFGpYKvx8pgAmEwz2EFTvfVc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '63133ca92c299bd23f40f94d370cf5381b0b1313', 'timestamp': '2021-09-16_17:11:22', 'tube': 'smithi', 'user': 'ivancich', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-17T01:55:51.865 DEBUG:teuthology.orchestra.run.smithi013:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-17T01:55:51.890 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-17T01:55:51.906 DEBUG:teuthology.orchestra.run.smithi013:> sudo install -d -m0777 -- /var/run/ceph 2021-09-17T01:55:51.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/run/ceph 2021-09-17T01:55:51.974 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:55:51.975 DEBUG:teuthology.orchestra.run.smithi013:> dd if=/scratch_devs of=/dev/stdout 2021-09-17T01:55:52.031 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-17T01:55:52.032 DEBUG:teuthology.orchestra.run.smithi013:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-17T01:55:52.088 INFO:teuthology.orchestra.run.smithi013.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-17T01:55:52.088 INFO:teuthology.orchestra.run.smithi013.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T01:55:52.088 INFO:teuthology.orchestra.run.smithi013.stdout:Device: 6h/6d Inode: 112433 Links: 1 2021-09-17T01:55:52.088 INFO:teuthology.orchestra.run.smithi013.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T01:55:52.089 INFO:teuthology.orchestra.run.smithi013.stdout:Context: system_u:object_r:device_t:s0 2021-09-17T01:55:52.089 INFO:teuthology.orchestra.run.smithi013.stdout:Access: 2021-09-17 01:55:02.264264890 +0000 2021-09-17T01:55:52.089 INFO:teuthology.orchestra.run.smithi013.stdout:Modify: 2021-09-17 01:45:30.420448554 +0000 2021-09-17T01:55:52.089 INFO:teuthology.orchestra.run.smithi013.stdout:Change: 2021-09-17 01:45:30.420448554 +0000 2021-09-17T01:55:52.090 INFO:teuthology.orchestra.run.smithi013.stdout: Birth: - 2021-09-17T01:55:52.098 INFO:teuthology.orchestra.run.smithi013.stderr:1+0 records in 2021-09-17T01:55:52.099 INFO:teuthology.orchestra.run.smithi013.stderr:1+0 records out 2021-09-17T01:55:52.099 INFO:teuthology.orchestra.run.smithi013.stderr:512 bytes copied, 0.000111164 s, 4.6 MB/s 2021-09-17T01:55:52.103 DEBUG:teuthology.orchestra.run.smithi013:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-17T01:55:52.159 INFO:teuthology.orchestra.run.smithi013.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-17T01:55:52.160 INFO:teuthology.orchestra.run.smithi013.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T01:55:52.160 INFO:teuthology.orchestra.run.smithi013.stdout:Device: 6h/6d Inode: 115077 Links: 1 2021-09-17T01:55:52.160 INFO:teuthology.orchestra.run.smithi013.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T01:55:52.160 INFO:teuthology.orchestra.run.smithi013.stdout:Context: system_u:object_r:device_t:s0 2021-09-17T01:55:52.160 INFO:teuthology.orchestra.run.smithi013.stdout:Access: 2021-09-17 01:55:02.264264890 +0000 2021-09-17T01:55:52.161 INFO:teuthology.orchestra.run.smithi013.stdout:Modify: 2021-09-17 01:45:30.770444442 +0000 2021-09-17T01:55:52.161 INFO:teuthology.orchestra.run.smithi013.stdout:Change: 2021-09-17 01:45:30.770444442 +0000 2021-09-17T01:55:52.161 INFO:teuthology.orchestra.run.smithi013.stdout: Birth: - 2021-09-17T01:55:52.169 INFO:teuthology.orchestra.run.smithi013.stderr:1+0 records in 2021-09-17T01:55:52.170 INFO:teuthology.orchestra.run.smithi013.stderr:1+0 records out 2021-09-17T01:55:52.170 INFO:teuthology.orchestra.run.smithi013.stderr:512 bytes copied, 0.000107042 s, 4.8 MB/s 2021-09-17T01:55:52.172 DEBUG:teuthology.orchestra.run.smithi013:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-17T01:55:52.228 INFO:teuthology.orchestra.run.smithi013.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-17T01:55:52.229 INFO:teuthology.orchestra.run.smithi013.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T01:55:52.229 INFO:teuthology.orchestra.run.smithi013.stdout:Device: 6h/6d Inode: 115908 Links: 1 2021-09-17T01:55:52.229 INFO:teuthology.orchestra.run.smithi013.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T01:55:52.229 INFO:teuthology.orchestra.run.smithi013.stdout:Context: system_u:object_r:device_t:s0 2021-09-17T01:55:52.229 INFO:teuthology.orchestra.run.smithi013.stdout:Access: 2021-09-17 01:55:02.264264890 +0000 2021-09-17T01:55:52.230 INFO:teuthology.orchestra.run.smithi013.stdout:Modify: 2021-09-17 01:45:31.098440590 +0000 2021-09-17T01:55:52.230 INFO:teuthology.orchestra.run.smithi013.stdout:Change: 2021-09-17 01:45:31.098440590 +0000 2021-09-17T01:55:52.230 INFO:teuthology.orchestra.run.smithi013.stdout: Birth: - 2021-09-17T01:55:52.238 INFO:teuthology.orchestra.run.smithi013.stderr:1+0 records in 2021-09-17T01:55:52.238 INFO:teuthology.orchestra.run.smithi013.stderr:1+0 records out 2021-09-17T01:55:52.239 INFO:teuthology.orchestra.run.smithi013.stderr:512 bytes copied, 0.000103948 s, 4.9 MB/s 2021-09-17T01:55:52.241 DEBUG:teuthology.orchestra.run.smithi013:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-17T01:55:52.297 INFO:teuthology.orchestra.run.smithi013.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-17T01:55:52.297 INFO:teuthology.orchestra.run.smithi013.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T01:55:52.297 INFO:teuthology.orchestra.run.smithi013.stdout:Device: 6h/6d Inode: 116818 Links: 1 2021-09-17T01:55:52.298 INFO:teuthology.orchestra.run.smithi013.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T01:55:52.298 INFO:teuthology.orchestra.run.smithi013.stdout:Context: system_u:object_r:device_t:s0 2021-09-17T01:55:52.298 INFO:teuthology.orchestra.run.smithi013.stdout:Access: 2021-09-17 01:55:02.264264890 +0000 2021-09-17T01:55:52.298 INFO:teuthology.orchestra.run.smithi013.stdout:Modify: 2021-09-17 01:45:31.440436572 +0000 2021-09-17T01:55:52.298 INFO:teuthology.orchestra.run.smithi013.stdout:Change: 2021-09-17 01:45:31.440436572 +0000 2021-09-17T01:55:52.299 INFO:teuthology.orchestra.run.smithi013.stdout: Birth: - 2021-09-17T01:55:52.306 INFO:teuthology.orchestra.run.smithi013.stderr:1+0 records in 2021-09-17T01:55:52.307 INFO:teuthology.orchestra.run.smithi013.stderr:1+0 records out 2021-09-17T01:55:52.307 INFO:teuthology.orchestra.run.smithi013.stderr:512 bytes copied, 8.599e-05 s, 6.0 MB/s 2021-09-17T01:55:52.309 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T01:55:52.309 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi013.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-17T01:55:52.310 INFO:tasks.ceph:Generating config... 2021-09-17T01:55:52.311 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-17T01:55:52.312 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-17T01:55:52.312 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-17T01:55:52.312 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T01:55:52.313 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-17T01:55:52.313 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-17T01:55:52.313 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-17T01:55:52.313 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-17T01:55:52.314 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-17T01:55:52.314 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-17T01:55:52.314 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-17T01:55:52.315 INFO:tasks.ceph:[client] setuser = ceph 2021-09-17T01:55:52.315 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-17T01:55:52.315 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-17T01:55:52.315 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-17T01:55:52.316 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-17T01:55:52.316 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-17T01:55:52.316 INFO:tasks.ceph:[osd] debug ms = 1 2021-09-17T01:55:52.317 INFO:tasks.ceph:[osd] debug osd = 20 2021-09-17T01:55:52.317 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-17T01:55:52.317 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-17T01:55:52.318 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-17T01:55:52.318 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-17T01:55:52.319 INFO:tasks.ceph:Setting up c1.mon.a... 2021-09-17T01:55:52.319 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-09-17T01:55:52.394 INFO:teuthology.orchestra.run.smithi013.stdout:creating /etc/ceph/c1.keyring 2021-09-17T01:55:52.397 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-09-17T01:55:52.436 DEBUG:teuthology.orchestra.run.smithi013:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-17T01:55:52.501 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.13')] 2021-09-17T01:55:52.502 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.13'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-17T01:55:52.503 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:55:52.503 DEBUG:teuthology.orchestra.run.smithi013:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-17T01:55:52.556 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.13 --print /home/ubuntu/cephtest/c1.monmap 2021-09-17T01:55:52.631 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:55:52.631 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:55:52.633 INFO:teuthology.orchestra.run.smithi013.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-09-17T01:55:52.633 INFO:teuthology.orchestra.run.smithi013.stdout:monmaptool: generated fsid 18deeb25-9bb1-48d4-a93a-0bdf7fba8ebb 2021-09-17T01:55:52.633 INFO:teuthology.orchestra.run.smithi013.stdout:setting min_mon_release = octopus 2021-09-17T01:55:52.633 INFO:teuthology.orchestra.run.smithi013.stdout:epoch 0 2021-09-17T01:55:52.634 INFO:teuthology.orchestra.run.smithi013.stdout:fsid 18deeb25-9bb1-48d4-a93a-0bdf7fba8ebb 2021-09-17T01:55:52.634 INFO:teuthology.orchestra.run.smithi013.stdout:last_changed 2021-09-17T01:55:52.631778+0000 2021-09-17T01:55:52.634 INFO:teuthology.orchestra.run.smithi013.stdout:created 2021-09-17T01:55:52.631778+0000 2021-09-17T01:55:52.634 INFO:teuthology.orchestra.run.smithi013.stdout:min_mon_release 15 (octopus) 2021-09-17T01:55:52.634 INFO:teuthology.orchestra.run.smithi013.stdout:election_strategy: 1 2021-09-17T01:55:52.635 INFO:teuthology.orchestra.run.smithi013.stdout:0: [v2:172.21.15.13:3300/0,v1:172.21.15.13:6789/0] mon.a 2021-09-17T01:55:52.635 INFO:teuthology.orchestra.run.smithi013.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-09-17T01:55:52.636 DEBUG:teuthology.orchestra.run.smithi013:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-17T01:55:52.688 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 18deeb25-9bb1-48d4-a93a-0bdf7fba8ebb... 2021-09-17T01:55:52.689 DEBUG:teuthology.orchestra.run.smithi013:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-17T01:55:52.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-17T01:55:52.772 INFO:teuthology.orchestra.run.smithi013.stdout:[global] 2021-09-17T01:55:52.772 INFO:teuthology.orchestra.run.smithi013.stdout: chdir = "" 2021-09-17T01:55:52.772 INFO:teuthology.orchestra.run.smithi013.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-17T01:55:52.773 INFO:teuthology.orchestra.run.smithi013.stdout: auth supported = cephx 2021-09-17T01:55:52.773 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.773 INFO:teuthology.orchestra.run.smithi013.stdout: filestore xattr use omap = true 2021-09-17T01:55:52.773 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.774 INFO:teuthology.orchestra.run.smithi013.stdout: mon clock drift allowed = 1.000 2021-09-17T01:55:52.774 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.774 INFO:teuthology.orchestra.run.smithi013.stdout: osd crush chooseleaf type = 0 2021-09-17T01:55:52.774 INFO:teuthology.orchestra.run.smithi013.stdout: auth debug = true 2021-09-17T01:55:52.774 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.775 INFO:teuthology.orchestra.run.smithi013.stdout: ms die on old message = true 2021-09-17T01:55:52.775 INFO:teuthology.orchestra.run.smithi013.stdout: ms die on bug = true 2021-09-17T01:55:52.775 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.775 INFO:teuthology.orchestra.run.smithi013.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-17T01:55:52.775 INFO:teuthology.orchestra.run.smithi013.stdout: mon pg warn max object skew = 0 2021-09-17T01:55:52.776 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.776 INFO:teuthology.orchestra.run.smithi013.stdout: # disable pg_autoscaler by default for new pools 2021-09-17T01:55:52.776 INFO:teuthology.orchestra.run.smithi013.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-17T01:55:52.776 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.777 INFO:teuthology.orchestra.run.smithi013.stdout: osd pool default size = 2 2021-09-17T01:55:52.777 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.777 INFO:teuthology.orchestra.run.smithi013.stdout: mon osd allow primary affinity = true 2021-09-17T01:55:52.777 INFO:teuthology.orchestra.run.smithi013.stdout: mon osd allow pg remap = true 2021-09-17T01:55:52.777 INFO:teuthology.orchestra.run.smithi013.stdout: mon warn on legacy crush tunables = false 2021-09-17T01:55:52.778 INFO:teuthology.orchestra.run.smithi013.stdout: mon warn on crush straw calc version zero = false 2021-09-17T01:55:52.778 INFO:teuthology.orchestra.run.smithi013.stdout: mon warn on no sortbitwise = false 2021-09-17T01:55:52.778 INFO:teuthology.orchestra.run.smithi013.stdout: mon warn on osd down out interval zero = false 2021-09-17T01:55:52.778 INFO:teuthology.orchestra.run.smithi013.stdout: mon warn on too few osds = false 2021-09-17T01:55:52.779 INFO:teuthology.orchestra.run.smithi013.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-17T01:55:52.779 INFO:teuthology.orchestra.run.smithi013.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-17T01:55:52.779 INFO:teuthology.orchestra.run.smithi013.stdout: mon_allow_pool_size_one = true 2021-09-17T01:55:52.779 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.779 INFO:teuthology.orchestra.run.smithi013.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-17T01:55:52.780 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.780 INFO:teuthology.orchestra.run.smithi013.stdout: osd default data pool replay window = 5 2021-09-17T01:55:52.780 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.780 INFO:teuthology.orchestra.run.smithi013.stdout: mon allow pool delete = true 2021-09-17T01:55:52.780 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.781 INFO:teuthology.orchestra.run.smithi013.stdout: mon cluster log file level = debug 2021-09-17T01:55:52.781 INFO:teuthology.orchestra.run.smithi013.stdout: debug asserts on shutdown = true 2021-09-17T01:55:52.781 INFO:teuthology.orchestra.run.smithi013.stdout: mon health detail to clog = false 2021-09-17T01:55:52.781 INFO:teuthology.orchestra.run.smithi013.stdout: mon host = 172.21.15.13 2021-09-17T01:55:52.782 INFO:teuthology.orchestra.run.smithi013.stdout: fsid = 18deeb25-9bb1-48d4-a93a-0bdf7fba8ebb 2021-09-17T01:55:52.782 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.782 INFO:teuthology.orchestra.run.smithi013.stdout:[osd] 2021-09-17T01:55:52.782 INFO:teuthology.orchestra.run.smithi013.stdout: osd journal size = 100 2021-09-17T01:55:52.782 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.783 INFO:teuthology.orchestra.run.smithi013.stdout: osd scrub load threshold = 5.0 2021-09-17T01:55:52.783 INFO:teuthology.orchestra.run.smithi013.stdout: osd scrub max interval = 600 2021-09-17T01:55:52.783 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.783 INFO:teuthology.orchestra.run.smithi013.stdout: osd recover clone overlap = true 2021-09-17T01:55:52.784 INFO:teuthology.orchestra.run.smithi013.stdout: osd recovery max chunk = 1048576 2021-09-17T01:55:52.784 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.784 INFO:teuthology.orchestra.run.smithi013.stdout: osd debug shutdown = true 2021-09-17T01:55:52.784 INFO:teuthology.orchestra.run.smithi013.stdout: osd debug op order = true 2021-09-17T01:55:52.784 INFO:teuthology.orchestra.run.smithi013.stdout: osd debug verify stray on activate = true 2021-09-17T01:55:52.785 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.785 INFO:teuthology.orchestra.run.smithi013.stdout: osd open classes on start = true 2021-09-17T01:55:52.785 INFO:teuthology.orchestra.run.smithi013.stdout: osd debug pg log writeout = true 2021-09-17T01:55:52.785 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.785 INFO:teuthology.orchestra.run.smithi013.stdout: osd deep scrub update digest min age = 30 2021-09-17T01:55:52.786 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.786 INFO:teuthology.orchestra.run.smithi013.stdout: osd map max advance = 10 2021-09-17T01:55:52.786 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.786 INFO:teuthology.orchestra.run.smithi013.stdout: journal zero on create = true 2021-09-17T01:55:52.787 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.787 INFO:teuthology.orchestra.run.smithi013.stdout: filestore ondisk finisher threads = 3 2021-09-17T01:55:52.787 INFO:teuthology.orchestra.run.smithi013.stdout: filestore apply finisher threads = 3 2021-09-17T01:55:52.787 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.787 INFO:teuthology.orchestra.run.smithi013.stdout: bdev debug aio = true 2021-09-17T01:55:52.788 INFO:teuthology.orchestra.run.smithi013.stdout: osd debug misdirected ops = true 2021-09-17T01:55:52.788 INFO:teuthology.orchestra.run.smithi013.stdout: debug ms = 1 2021-09-17T01:55:52.788 INFO:teuthology.orchestra.run.smithi013.stdout: debug osd = 20 2021-09-17T01:55:52.788 INFO:teuthology.orchestra.run.smithi013.stdout: rocksdb delete range threshold = 0 2021-09-17T01:55:52.788 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.789 INFO:teuthology.orchestra.run.smithi013.stdout:[mgr] 2021-09-17T01:55:52.789 INFO:teuthology.orchestra.run.smithi013.stdout: debug ms = 1 2021-09-17T01:55:52.789 INFO:teuthology.orchestra.run.smithi013.stdout: debug mgr = 20 2021-09-17T01:55:52.789 INFO:teuthology.orchestra.run.smithi013.stdout: debug mon = 20 2021-09-17T01:55:52.789 INFO:teuthology.orchestra.run.smithi013.stdout: debug auth = 20 2021-09-17T01:55:52.790 INFO:teuthology.orchestra.run.smithi013.stdout: mon reweight min pgs per osd = 4 2021-09-17T01:55:52.790 INFO:teuthology.orchestra.run.smithi013.stdout: mon reweight min bytes per osd = 10 2021-09-17T01:55:52.790 INFO:teuthology.orchestra.run.smithi013.stdout: mgr/telemetry/nag = false 2021-09-17T01:55:52.790 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.790 INFO:teuthology.orchestra.run.smithi013.stdout:[mon] 2021-09-17T01:55:52.791 INFO:teuthology.orchestra.run.smithi013.stdout: debug ms = 1 2021-09-17T01:55:52.791 INFO:teuthology.orchestra.run.smithi013.stdout: debug mon = 20 2021-09-17T01:55:52.791 INFO:teuthology.orchestra.run.smithi013.stdout: debug paxos = 20 2021-09-17T01:55:52.791 INFO:teuthology.orchestra.run.smithi013.stdout: debug auth = 20 2021-09-17T01:55:52.791 INFO:teuthology.orchestra.run.smithi013.stdout: mon data avail warn = 5 2021-09-17T01:55:52.792 INFO:teuthology.orchestra.run.smithi013.stdout: mon mgr mkfs grace = 240 2021-09-17T01:55:52.792 INFO:teuthology.orchestra.run.smithi013.stdout: mon reweight min pgs per osd = 4 2021-09-17T01:55:52.792 INFO:teuthology.orchestra.run.smithi013.stdout: mon osd reporter subtree level = osd 2021-09-17T01:55:52.792 INFO:teuthology.orchestra.run.smithi013.stdout: mon osd prime pg temp = true 2021-09-17T01:55:52.792 INFO:teuthology.orchestra.run.smithi013.stdout: mon reweight min bytes per osd = 10 2021-09-17T01:55:52.793 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.793 INFO:teuthology.orchestra.run.smithi013.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-17T01:55:52.793 INFO:teuthology.orchestra.run.smithi013.stdout: auth mon ticket ttl = 660 # 11m 2021-09-17T01:55:52.793 INFO:teuthology.orchestra.run.smithi013.stdout: auth service ticket ttl = 240 # 4m 2021-09-17T01:55:52.794 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.794 INFO:teuthology.orchestra.run.smithi013.stdout: # don't complain about insecure global_id in the test suite 2021-09-17T01:55:52.794 INFO:teuthology.orchestra.run.smithi013.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-17T01:55:52.794 INFO:teuthology.orchestra.run.smithi013.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-17T01:55:52.794 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:55:52.795 INFO:teuthology.orchestra.run.smithi013.stdout:[client] 2021-09-17T01:55:52.795 INFO:teuthology.orchestra.run.smithi013.stdout: rgw cache enabled = true 2021-09-17T01:55:52.795 INFO:teuthology.orchestra.run.smithi013.stdout: rgw enable ops log = true 2021-09-17T01:55:52.795 INFO:teuthology.orchestra.run.smithi013.stdout: rgw enable usage log = true 2021-09-17T01:55:52.795 INFO:teuthology.orchestra.run.smithi013.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-17T01:55:52.796 INFO:teuthology.orchestra.run.smithi013.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-17T01:55:52.796 INFO:teuthology.orchestra.run.smithi013.stdout: debug rgw = 20 2021-09-17T01:55:52.796 INFO:teuthology.orchestra.run.smithi013.stdout: rgw crypt require ssl = False 2021-09-17T01:55:52.796 INFO:teuthology.orchestra.run.smithi013.stdout: rgw crypt s3 kms backend = testing 2021-09-17T01:55:52.797 INFO:teuthology.orchestra.run.smithi013.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T01:55:52.797 INFO:teuthology.orchestra.run.smithi013.stdout: rgw curl low speed time = 300 2021-09-17T01:55:52.797 INFO:teuthology.orchestra.run.smithi013.stdout: rgw data log num shards = 4 2021-09-17T01:55:52.797 INFO:teuthology.orchestra.run.smithi013.stdout: rgw md log max shards = 4 2021-09-17T01:55:52.797 INFO:teuthology.orchestra.run.smithi013.stdout: rgw sync log trim interval = 0 2021-09-17T01:55:52.798 INFO:teuthology.orchestra.run.smithi013.stdout: rgw sync meta inject err probability = 0.1 2021-09-17T01:55:52.798 INFO:teuthology.orchestra.run.smithi013.stdout: rgw sync obj etag verify = True 2021-09-17T01:55:52.798 INFO:teuthology.orchestra.run.smithi013.stdout: setgroup = ceph 2021-09-17T01:55:52.798 INFO:teuthology.orchestra.run.smithi013.stdout: setuser = ceph 2021-09-17T01:55:52.799 INFO:teuthology.orchestra.run.smithi013.stdout:[mon.a] 2021-09-17T01:55:52.799 INFO:teuthology.orchestra.run.smithi013.stdout:[osd.0] 2021-09-17T01:55:52.799 INFO:teuthology.orchestra.run.smithi013.stdout: osd_max_omap_entries_per_request = 10 2021-09-17T01:55:52.800 INFO:teuthology.orchestra.run.smithi013.stdout:[osd.1] 2021-09-17T01:55:52.800 INFO:teuthology.orchestra.run.smithi013.stdout: osd_max_omap_entries_per_request = 1000 2021-09-17T01:55:52.800 INFO:teuthology.orchestra.run.smithi013.stdout:[osd.2] 2021-09-17T01:55:52.800 INFO:teuthology.orchestra.run.smithi013.stdout: osd_max_omap_entries_per_request = 10000 2021-09-17T01:55:52.802 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2021-09-17T01:55:52.802 INFO:teuthology.orchestra.run.smithi045.stdout: chdir = "" 2021-09-17T01:55:52.803 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-17T01:55:52.803 INFO:teuthology.orchestra.run.smithi045.stdout: auth supported = cephx 2021-09-17T01:55:52.803 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.803 INFO:teuthology.orchestra.run.smithi045.stdout: filestore xattr use omap = true 2021-09-17T01:55:52.804 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.804 INFO:teuthology.orchestra.run.smithi045.stdout: mon clock drift allowed = 1.000 2021-09-17T01:55:52.804 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.804 INFO:teuthology.orchestra.run.smithi045.stdout: osd crush chooseleaf type = 0 2021-09-17T01:55:52.805 INFO:teuthology.orchestra.run.smithi045.stdout: auth debug = true 2021-09-17T01:55:52.805 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.805 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on old message = true 2021-09-17T01:55:52.805 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on bug = true 2021-09-17T01:55:52.806 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.806 INFO:teuthology.orchestra.run.smithi045.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-17T01:55:52.806 INFO:teuthology.orchestra.run.smithi045.stdout: mon pg warn max object skew = 0 2021-09-17T01:55:52.806 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.806 INFO:teuthology.orchestra.run.smithi045.stdout: # disable pg_autoscaler by default for new pools 2021-09-17T01:55:52.807 INFO:teuthology.orchestra.run.smithi045.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-17T01:55:52.807 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.807 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default size = 2 2021-09-17T01:55:52.807 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.807 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow primary affinity = true 2021-09-17T01:55:52.808 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow pg remap = true 2021-09-17T01:55:52.808 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on legacy crush tunables = false 2021-09-17T01:55:52.808 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on crush straw calc version zero = false 2021-09-17T01:55:52.808 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on no sortbitwise = false 2021-09-17T01:55:52.809 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on osd down out interval zero = false 2021-09-17T01:55:52.809 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on too few osds = false 2021-09-17T01:55:52.809 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-17T01:55:52.809 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-17T01:55:52.809 INFO:teuthology.orchestra.run.smithi045.stdout: mon_allow_pool_size_one = true 2021-09-17T01:55:52.810 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.810 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-17T01:55:52.812 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.812 INFO:teuthology.orchestra.run.smithi045.stdout: osd default data pool replay window = 5 2021-09-17T01:55:52.812 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.812 INFO:teuthology.orchestra.run.smithi045.stdout: mon allow pool delete = true 2021-09-17T01:55:52.813 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.813 INFO:teuthology.orchestra.run.smithi045.stdout: mon cluster log file level = debug 2021-09-17T01:55:52.813 INFO:teuthology.orchestra.run.smithi045.stdout: debug asserts on shutdown = true 2021-09-17T01:55:52.813 INFO:teuthology.orchestra.run.smithi045.stdout: mon health detail to clog = false 2021-09-17T01:55:52.813 INFO:teuthology.orchestra.run.smithi045.stdout: mon host = 172.21.15.13 2021-09-17T01:55:52.814 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = 18deeb25-9bb1-48d4-a93a-0bdf7fba8ebb 2021-09-17T01:55:52.814 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.814 INFO:teuthology.orchestra.run.smithi045.stdout:[osd] 2021-09-17T01:55:52.814 INFO:teuthology.orchestra.run.smithi045.stdout: osd journal size = 100 2021-09-17T01:55:52.815 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.815 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub load threshold = 5.0 2021-09-17T01:55:52.815 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub max interval = 600 2021-09-17T01:55:52.815 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.816 INFO:teuthology.orchestra.run.smithi045.stdout: osd recover clone overlap = true 2021-09-17T01:55:52.816 INFO:teuthology.orchestra.run.smithi045.stdout: osd recovery max chunk = 1048576 2021-09-17T01:55:52.816 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.816 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug shutdown = true 2021-09-17T01:55:52.816 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug op order = true 2021-09-17T01:55:52.817 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify stray on activate = true 2021-09-17T01:55:52.817 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.817 INFO:teuthology.orchestra.run.smithi045.stdout: osd open classes on start = true 2021-09-17T01:55:52.817 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug pg log writeout = true 2021-09-17T01:55:52.818 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.818 INFO:teuthology.orchestra.run.smithi045.stdout: osd deep scrub update digest min age = 30 2021-09-17T01:55:52.818 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.818 INFO:teuthology.orchestra.run.smithi045.stdout: osd map max advance = 10 2021-09-17T01:55:52.819 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.819 INFO:teuthology.orchestra.run.smithi045.stdout: journal zero on create = true 2021-09-17T01:55:52.819 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.819 INFO:teuthology.orchestra.run.smithi045.stdout: filestore ondisk finisher threads = 3 2021-09-17T01:55:52.819 INFO:teuthology.orchestra.run.smithi045.stdout: filestore apply finisher threads = 3 2021-09-17T01:55:52.820 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.820 INFO:teuthology.orchestra.run.smithi045.stdout: bdev debug aio = true 2021-09-17T01:55:52.820 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug misdirected ops = true 2021-09-17T01:55:52.820 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2021-09-17T01:55:52.821 INFO:teuthology.orchestra.run.smithi045.stdout: debug osd = 20 2021-09-17T01:55:52.821 INFO:teuthology.orchestra.run.smithi045.stdout: rocksdb delete range threshold = 0 2021-09-17T01:55:52.821 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.821 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr] 2021-09-17T01:55:52.821 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2021-09-17T01:55:52.822 INFO:teuthology.orchestra.run.smithi045.stdout: debug mgr = 20 2021-09-17T01:55:52.822 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2021-09-17T01:55:52.822 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2021-09-17T01:55:52.822 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2021-09-17T01:55:52.823 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2021-09-17T01:55:52.823 INFO:teuthology.orchestra.run.smithi045.stdout: mgr/telemetry/nag = false 2021-09-17T01:55:52.823 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.823 INFO:teuthology.orchestra.run.smithi045.stdout:[mon] 2021-09-17T01:55:52.823 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2021-09-17T01:55:52.824 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2021-09-17T01:55:52.824 INFO:teuthology.orchestra.run.smithi045.stdout: debug paxos = 20 2021-09-17T01:55:52.824 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2021-09-17T01:55:52.824 INFO:teuthology.orchestra.run.smithi045.stdout: mon data avail warn = 5 2021-09-17T01:55:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: mon mgr mkfs grace = 240 2021-09-17T01:55:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2021-09-17T01:55:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd reporter subtree level = osd 2021-09-17T01:55:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd prime pg temp = true 2021-09-17T01:55:52.825 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2021-09-17T01:55:52.826 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.826 INFO:teuthology.orchestra.run.smithi045.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-17T01:55:52.826 INFO:teuthology.orchestra.run.smithi045.stdout: auth mon ticket ttl = 660 # 11m 2021-09-17T01:55:52.826 INFO:teuthology.orchestra.run.smithi045.stdout: auth service ticket ttl = 240 # 4m 2021-09-17T01:55:52.827 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.827 INFO:teuthology.orchestra.run.smithi045.stdout: # don't complain about insecure global_id in the test suite 2021-09-17T01:55:52.827 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-17T01:55:52.827 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-17T01:55:52.828 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:55:52.828 INFO:teuthology.orchestra.run.smithi045.stdout:[client] 2021-09-17T01:55:52.828 INFO:teuthology.orchestra.run.smithi045.stdout: rgw cache enabled = true 2021-09-17T01:55:52.828 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable ops log = true 2021-09-17T01:55:52.828 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable usage log = true 2021-09-17T01:55:52.829 INFO:teuthology.orchestra.run.smithi045.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-17T01:55:52.829 INFO:teuthology.orchestra.run.smithi045.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-17T01:55:52.829 INFO:teuthology.orchestra.run.smithi045.stdout: debug rgw = 20 2021-09-17T01:55:52.829 INFO:teuthology.orchestra.run.smithi045.stdout: rgw crypt require ssl = False 2021-09-17T01:55:52.829 INFO:teuthology.orchestra.run.smithi045.stdout: rgw crypt s3 kms backend = testing 2021-09-17T01:55:52.830 INFO:teuthology.orchestra.run.smithi045.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T01:55:52.830 INFO:teuthology.orchestra.run.smithi045.stdout: rgw curl low speed time = 300 2021-09-17T01:55:52.830 INFO:teuthology.orchestra.run.smithi045.stdout: rgw data log num shards = 4 2021-09-17T01:55:52.830 INFO:teuthology.orchestra.run.smithi045.stdout: rgw md log max shards = 4 2021-09-17T01:55:52.831 INFO:teuthology.orchestra.run.smithi045.stdout: rgw sync log trim interval = 0 2021-09-17T01:55:52.831 INFO:teuthology.orchestra.run.smithi045.stdout: rgw sync meta inject err probability = 0.1 2021-09-17T01:55:52.831 INFO:teuthology.orchestra.run.smithi045.stdout: rgw sync obj etag verify = True 2021-09-17T01:55:52.831 INFO:teuthology.orchestra.run.smithi045.stdout: setgroup = ceph 2021-09-17T01:55:52.832 INFO:teuthology.orchestra.run.smithi045.stdout: setuser = ceph 2021-09-17T01:55:52.832 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.a] 2021-09-17T01:55:52.832 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.0] 2021-09-17T01:55:52.832 INFO:teuthology.orchestra.run.smithi045.stdout: osd_max_omap_entries_per_request = 10 2021-09-17T01:55:52.832 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.1] 2021-09-17T01:55:52.833 INFO:teuthology.orchestra.run.smithi045.stdout: osd_max_omap_entries_per_request = 1000 2021-09-17T01:55:52.833 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.2] 2021-09-17T01:55:52.833 INFO:teuthology.orchestra.run.smithi045.stdout: osd_max_omap_entries_per_request = 10000 2021-09-17T01:55:52.833 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-09-17T01:55:52.834 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-09-17T01:55:52.893 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-17T01:55:52.893 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:55:52.893 DEBUG:teuthology.orchestra.run.smithi013:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2021-09-17T01:55:52.909 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:55:52.910 DEBUG:teuthology.orchestra.run.smithi013:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2021-09-17T01:55:52.964 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi013.front.sepia.ceph.com 2021-09-17T01:55:52.964 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:55:52.965 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd of=/etc/ceph/c1.keyring 2021-09-17T01:55:52.965 DEBUG:teuthology.orchestra.run.smithi013:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-17T01:55:53.039 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:55:53.039 DEBUG:teuthology.orchestra.run.smithi013:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-17T01:55:53.094 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi045.front.sepia.ceph.com 2021-09-17T01:55:53.095 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:55:53.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/c1.keyring 2021-09-17T01:55:53.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-17T01:55:53.137 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:55:53.137 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-17T01:55:53.195 INFO:tasks.ceph:Setting up mon nodes... 2021-09-17T01:55:53.195 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-17T01:55:53.196 DEBUG:teuthology.orchestra.run.smithi013:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-09-17T01:55:53.244 INFO:teuthology.orchestra.run.smithi013.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-09-17T01:55:53.246 INFO:tasks.ceph:Setting up mds nodes... 2021-09-17T01:55:53.247 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-17T01:55:53.247 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-09-17T01:55:53.285 INFO:teuthology.orchestra.run.smithi013.stdout:creating /etc/ceph/c1.client.0.keyring 2021-09-17T01:55:53.297 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-09-17T01:55:53.374 INFO:teuthology.orchestra.run.smithi013.stdout:creating /etc/ceph/c1.client.1.keyring 2021-09-17T01:55:53.387 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-17T01:55:53.388 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi013.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-17T01:55:53.388 DEBUG:teuthology.orchestra.run.smithi013:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-09-17T01:55:53.452 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T01:55:53.452 INFO:tasks.ceph:role: c1.osd.0 2021-09-17T01:55:53.452 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi013.front.sepia.ceph.com 2021-09-17T01:55:53.453 DEBUG:teuthology.orchestra.run.smithi013:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-17T01:55:53.517 INFO:teuthology.orchestra.run.smithi013.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T01:55:53.517 INFO:teuthology.orchestra.run.smithi013.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T01:55:53.517 INFO:teuthology.orchestra.run.smithi013.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T01:55:53.517 INFO:teuthology.orchestra.run.smithi013.stdout: = reflink=1 2021-09-17T01:55:53.518 INFO:teuthology.orchestra.run.smithi013.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T01:55:53.518 INFO:teuthology.orchestra.run.smithi013.stdout: = sunit=0 swidth=0 blks 2021-09-17T01:55:53.518 INFO:teuthology.orchestra.run.smithi013.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T01:55:53.518 INFO:teuthology.orchestra.run.smithi013.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T01:55:53.519 INFO:teuthology.orchestra.run.smithi013.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T01:55:53.519 INFO:teuthology.orchestra.run.smithi013.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T01:55:53.539 INFO:teuthology.orchestra.run.smithi013.stdout:Discarding blocks...Done. 2021-09-17T01:55:53.540 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi013.front.sepia.ceph.com -o noatime 2021-09-17T01:55:53.541 DEBUG:teuthology.orchestra.run.smithi013:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-09-17T01:55:53.619 DEBUG:teuthology.orchestra.run.smithi013:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-09-17T01:55:53.688 DEBUG:teuthology.orchestra.run.smithi013:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-09-17T01:55:53.753 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T01:55:53.753 INFO:tasks.ceph:role: c1.osd.1 2021-09-17T01:55:53.754 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi013.front.sepia.ceph.com 2021-09-17T01:55:53.754 DEBUG:teuthology.orchestra.run.smithi013:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-17T01:55:53.817 INFO:teuthology.orchestra.run.smithi013.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T01:55:53.817 INFO:teuthology.orchestra.run.smithi013.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T01:55:53.818 INFO:teuthology.orchestra.run.smithi013.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T01:55:53.818 INFO:teuthology.orchestra.run.smithi013.stdout: = reflink=1 2021-09-17T01:55:53.818 INFO:teuthology.orchestra.run.smithi013.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T01:55:53.819 INFO:teuthology.orchestra.run.smithi013.stdout: = sunit=0 swidth=0 blks 2021-09-17T01:55:53.819 INFO:teuthology.orchestra.run.smithi013.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T01:55:53.819 INFO:teuthology.orchestra.run.smithi013.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T01:55:53.819 INFO:teuthology.orchestra.run.smithi013.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T01:55:53.819 INFO:teuthology.orchestra.run.smithi013.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T01:55:53.840 INFO:teuthology.orchestra.run.smithi013.stdout:Discarding blocks...Done. 2021-09-17T01:55:53.841 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi013.front.sepia.ceph.com -o noatime 2021-09-17T01:55:53.842 DEBUG:teuthology.orchestra.run.smithi013:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-09-17T01:55:53.920 DEBUG:teuthology.orchestra.run.smithi013:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-09-17T01:55:53.988 DEBUG:teuthology.orchestra.run.smithi013:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-09-17T01:55:54.053 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T01:55:54.053 INFO:tasks.ceph:role: c1.osd.2 2021-09-17T01:55:54.054 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi013.front.sepia.ceph.com 2021-09-17T01:55:54.054 DEBUG:teuthology.orchestra.run.smithi013:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-17T01:55:54.117 INFO:teuthology.orchestra.run.smithi013.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T01:55:54.117 INFO:teuthology.orchestra.run.smithi013.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T01:55:54.117 INFO:teuthology.orchestra.run.smithi013.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T01:55:54.118 INFO:teuthology.orchestra.run.smithi013.stdout: = reflink=1 2021-09-17T01:55:54.118 INFO:teuthology.orchestra.run.smithi013.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T01:55:54.118 INFO:teuthology.orchestra.run.smithi013.stdout: = sunit=0 swidth=0 blks 2021-09-17T01:55:54.118 INFO:teuthology.orchestra.run.smithi013.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T01:55:54.119 INFO:teuthology.orchestra.run.smithi013.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T01:55:54.119 INFO:teuthology.orchestra.run.smithi013.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T01:55:54.119 INFO:teuthology.orchestra.run.smithi013.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T01:55:54.139 INFO:teuthology.orchestra.run.smithi013.stdout:Discarding blocks...Done. 2021-09-17T01:55:54.141 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi013.front.sepia.ceph.com -o noatime 2021-09-17T01:55:54.141 DEBUG:teuthology.orchestra.run.smithi013:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-09-17T01:55:54.219 DEBUG:teuthology.orchestra.run.smithi013:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-09-17T01:55:54.296 DEBUG:teuthology.orchestra.run.smithi013:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-17T01:55:54.379 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:54.377+0000 7fc4a72e90c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-09-17T01:55:54.380 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:54.377+0000 7fc4a72e90c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-09-17T01:55:54.380 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:54.377+0000 7fc4a72e90c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-17T01:55:54.380 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:54.377+0000 7fc4a72e90c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-17T01:55:54.381 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:54.377+0000 7fc4a72e90c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-17T01:55:54.381 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:54.377+0000 7fc4a72e90c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-09-17T01:55:56.468 DEBUG:teuthology.orchestra.run.smithi013:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-09-17T01:55:56.538 DEBUG:teuthology.orchestra.run.smithi013:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-17T01:55:56.627 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:56.625+0000 7f86f0fa20c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-09-17T01:55:56.628 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:56.626+0000 7f86f0fa20c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-09-17T01:55:56.628 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:56.626+0000 7f86f0fa20c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-17T01:55:56.628 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:56.626+0000 7f86f0fa20c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-17T01:55:56.628 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:56.626+0000 7f86f0fa20c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-17T01:55:56.629 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:56.626+0000 7f86f0fa20c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-09-17T01:55:58.707 DEBUG:teuthology.orchestra.run.smithi013:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-09-17T01:55:58.737 DEBUG:teuthology.orchestra.run.smithi013:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-17T01:55:58.824 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:58.821+0000 7fc88a09d0c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-09-17T01:55:58.824 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:58.822+0000 7fc88a09d0c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-09-17T01:55:58.824 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:58.822+0000 7fc88a09d0c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-17T01:55:58.825 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:58.822+0000 7fc88a09d0c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-17T01:55:58.825 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:58.822+0000 7fc88a09d0c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-17T01:55:58.825 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:55:58.822+0000 7fc88a09d0c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-09-17T01:56:00.908 DEBUG:teuthology.orchestra.run.smithi013:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-09-17T01:56:00.937 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-17T01:56:00.938 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:56:00.938 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2021-09-17T01:56:01.006 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:56:01.006 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2021-09-17T01:56:01.073 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:56:01.073 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2021-09-17T01:56:01.140 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:56:01.141 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2021-09-17T01:56:01.207 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:56:01.207 DEBUG:teuthology.orchestra.run.smithi013:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2021-09-17T01:56:01.263 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:56:01.264 DEBUG:teuthology.orchestra.run.smithi013:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2021-09-17T01:56:01.319 INFO:tasks.ceph:Adding keys to all mons... 2021-09-17T01:56:01.319 DEBUG:teuthology.orchestra.run.smithi013:> sudo tee -a /etc/ceph/c1.keyring 2021-09-17T01:56:01.382 INFO:teuthology.orchestra.run.smithi013.stdout:[mgr.x] 2021-09-17T01:56:01.383 INFO:teuthology.orchestra.run.smithi013.stdout: key = AQCp9UNhCYSIDhAAwMH4SUuvuOjMT7siQyJNKg== 2021-09-17T01:56:01.383 INFO:teuthology.orchestra.run.smithi013.stdout:[osd.0] 2021-09-17T01:56:01.383 INFO:teuthology.orchestra.run.smithi013.stdout: key = AQCq9UNhPuSPFhAAy/4cdeMnTO6PpYpOvhHAaw== 2021-09-17T01:56:01.384 INFO:teuthology.orchestra.run.smithi013.stdout:[osd.1] 2021-09-17T01:56:01.384 INFO:teuthology.orchestra.run.smithi013.stdout: key = AQCs9UNhMyNiJRAAPlFJFEKZNVMqw0jL1GrvJw== 2021-09-17T01:56:01.384 INFO:teuthology.orchestra.run.smithi013.stdout:[osd.2] 2021-09-17T01:56:01.384 INFO:teuthology.orchestra.run.smithi013.stdout: key = AQCu9UNhZtQSMRAApFfOsoH0lzkvR8lBR53RoA== 2021-09-17T01:56:01.384 INFO:teuthology.orchestra.run.smithi013.stdout:[client.0] 2021-09-17T01:56:01.385 INFO:teuthology.orchestra.run.smithi013.stdout: key = AQCp9UNh7mgAERAAkVuuYPvPF8+y6FQYVYOj9g== 2021-09-17T01:56:01.385 INFO:teuthology.orchestra.run.smithi013.stdout:[client.1] 2021-09-17T01:56:01.385 INFO:teuthology.orchestra.run.smithi013.stdout: key = AQCp9UNha7xXFhAAri7rSXhAwTygB1bytjJFew== 2021-09-17T01:56:01.386 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-17T01:56:01.476 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T01:56:01.524 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T01:56:01.571 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T01:56:01.619 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-17T01:56:01.669 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-17T01:56:01.719 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-17T01:56:01.720 DEBUG:teuthology.orchestra.run.smithi013:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-09-17T01:56:01.745 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-09-17T01:56:01.837 DEBUG:teuthology.orchestra.run.smithi013:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-09-17T01:56:01.861 DEBUG:teuthology.orchestra.run.smithi013:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-09-17T01:56:01.916 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-09-17T01:56:01.916 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-09-17T01:56:01.917 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-09-17T01:56:01.959 INFO:tasks.ceph.c1.mon.a:Started 2021-09-17T01:56:01.959 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-09-17T01:56:01.960 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-09-17T01:56:01.960 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-09-17T01:56:01.962 INFO:tasks.ceph.c1.mgr.x:Started 2021-09-17T01:56:01.963 INFO:tasks.ceph:Setting crush tunables to default 2021-09-17T01:56:01.963 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph --cluster c1 osd crush tunables default 2021-09-17T01:56:02.163 INFO:teuthology.orchestra.run.smithi013.stderr:adjusted tunables profile to default 2021-09-17T01:56:02.174 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-09-17T01:56:02.174 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:56:02.175 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2021-09-17T01:56:02.199 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:56:02.199 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2021-09-17T01:56:02.264 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:56:02.265 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2021-09-17T01:56:02.329 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph --cluster c1 osd new 09578819-f455-465c-b992-5452e4821af8 0 2021-09-17T01:56:02.577 INFO:teuthology.orchestra.run.smithi013.stdout:0 2021-09-17T01:56:02.590 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph --cluster c1 osd new f61e2fdb-384d-4b4a-949c-d035f5b44df4 1 2021-09-17T01:56:02.815 INFO:teuthology.orchestra.run.smithi013.stdout:1 2021-09-17T01:56:02.827 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph --cluster c1 osd new fe68cba9-61ba-4070-9f71-acc3030ff781 2 2021-09-17T01:56:03.034 INFO:teuthology.orchestra.run.smithi013.stdout:2 2021-09-17T01:56:03.047 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-09-17T01:56:03.047 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-09-17T01:56:03.049 INFO:tasks.ceph.c1.osd.0:Started 2021-09-17T01:56:03.050 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-09-17T01:56:03.050 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-09-17T01:56:03.052 INFO:tasks.ceph.c1.osd.1:Started 2021-09-17T01:56:03.053 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-09-17T01:56:03.053 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-09-17T01:56:03.055 INFO:tasks.ceph.c1.osd.2:Started 2021-09-17T01:56:03.055 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-17T01:56:03.359 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-17T01:56:03.557 INFO:teuthology.misc.health.smithi013.stdout: 2021-09-17T01:56:03.557 INFO:teuthology.misc.health.smithi013.stdout:{"epoch":5,"fsid":"18deeb25-9bb1-48d4-a93a-0bdf7fba8ebb","created":"2021-09-17T01:56:02.057407+0000","modified":"2021-09-17T01:56:03.032931+0000","last_up_change":"0.000000","last_in_change":"2021-09-17T01:56:03.032931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"09578819-f455-465c-b992-5452e4821af8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f61e2fdb-384d-4b4a-949c-d035f5b44df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fe68cba9-61ba-4070-9f71-acc3030ff781","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T01:56:03.568 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-17T01:56:03.645 INFO:tasks.ceph.c1.osd.0.smithi013.stderr:2021-09-17T01:56:03.643+0000 7ff3848480c0 -1 Falling back to public interface 2021-09-17T01:56:03.647 INFO:tasks.ceph.c1.osd.1.smithi013.stderr:2021-09-17T01:56:03.645+0000 7f690ecf20c0 -1 Falling back to public interface 2021-09-17T01:56:03.654 INFO:tasks.ceph.c1.osd.2.smithi013.stderr:2021-09-17T01:56:03.652+0000 7f5692e430c0 -1 Falling back to public interface 2021-09-17T01:56:06.402 INFO:tasks.ceph.c1.osd.1.smithi013.stderr:2021-09-17T01:56:06.400+0000 7f690ecf20c0 -1 osd.1 0 log_to_monitors {default=true} 2021-09-17T01:56:06.403 INFO:tasks.ceph.c1.osd.0.smithi013.stderr:2021-09-17T01:56:06.401+0000 7ff3848480c0 -1 osd.0 0 log_to_monitors {default=true} 2021-09-17T01:56:06.409 INFO:tasks.ceph.c1.osd.2.smithi013.stderr:2021-09-17T01:56:06.408+0000 7f5692e430c0 -1 osd.2 0 log_to_monitors {default=true} 2021-09-17T01:56:08.392 INFO:tasks.ceph.c1.osd.2.smithi013.stderr:2021-09-17T01:56:08.390+0000 7f5689801700 -1 osd.2 0 waiting for initial osdmap 2021-09-17T01:56:08.393 INFO:tasks.ceph.c1.osd.2.smithi013.stderr:2021-09-17T01:56:08.392+0000 7f5682194700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T01:56:08.615 INFO:tasks.ceph.c1.osd.0.smithi013.stderr:2021-09-17T01:56:08.613+0000 7ff37ca09700 -1 osd.0 0 waiting for initial osdmap 2021-09-17T01:56:08.617 INFO:tasks.ceph.c1.osd.0.smithi013.stderr:2021-09-17T01:56:08.615+0000 7ff373b99700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T01:56:08.845 INFO:tasks.ceph.c1.osd.1.smithi013.stderr:2021-09-17T01:56:08.843+0000 7f69056b0700 -1 osd.1 0 waiting for initial osdmap 2021-09-17T01:56:08.846 INFO:tasks.ceph.c1.osd.1.smithi013.stderr:2021-09-17T01:56:08.845+0000 7f68fe043700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T01:56:09.872 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-17T01:56:10.184 INFO:teuthology.misc.health.smithi013.stdout: 2021-09-17T01:56:10.185 INFO:teuthology.misc.health.smithi013.stdout:{"epoch":8,"fsid":"18deeb25-9bb1-48d4-a93a-0bdf7fba8ebb","created":"2021-09-17T01:56:02.057407+0000","modified":"2021-09-17T01:56:09.389359+0000","last_up_change":"2021-09-17T01:56:09.389359+0000","last_in_change":"2021-09-17T01:56:03.032931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"09578819-f455-465c-b992-5452e4821af8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6800","nonce":31452},{"type":"v1","addr":"172.21.15.13:6801","nonce":31452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6802","nonce":31452},{"type":"v1","addr":"172.21.15.13:6803","nonce":31452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6806","nonce":31452},{"type":"v1","addr":"172.21.15.13:6807","nonce":31452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6804","nonce":31452},{"type":"v1","addr":"172.21.15.13:6805","nonce":31452}]},"public_addr":"172.21.15.13:6801/31452","cluster_addr":"172.21.15.13:6803/31452","heartbeat_back_addr":"172.21.15.13:6807/31452","heartbeat_front_addr":"172.21.15.13:6805/31452","state":["exists","up"]},{"osd":1,"uuid":"f61e2fdb-384d-4b4a-949c-d035f5b44df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6808","nonce":31453},{"type":"v1","addr":"172.21.15.13:6809","nonce":31453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6810","nonce":31453},{"type":"v1","addr":"172.21.15.13:6811","nonce":31453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6814","nonce":31453},{"type":"v1","addr":"172.21.15.13:6815","nonce":31453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6812","nonce":31453},{"type":"v1","addr":"172.21.15.13:6813","nonce":31453}]},"public_addr":"172.21.15.13:6809/31453","cluster_addr":"172.21.15.13:6811/31453","heartbeat_back_addr":"172.21.15.13:6815/31453","heartbeat_front_addr":"172.21.15.13:6813/31453","state":["exists","up"]},{"osd":2,"uuid":"fe68cba9-61ba-4070-9f71-acc3030ff781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6816","nonce":31454},{"type":"v1","addr":"172.21.15.13:6817","nonce":31454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6818","nonce":31454},{"type":"v1","addr":"172.21.15.13:6819","nonce":31454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6822","nonce":31454},{"type":"v1","addr":"172.21.15.13:6823","nonce":31454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6820","nonce":31454},{"type":"v1","addr":"172.21.15.13:6821","nonce":31454}]},"public_addr":"172.21.15.13:6817/31454","cluster_addr":"172.21.15.13:6819/31454","heartbeat_back_addr":"172.21.15.13:6823/31454","heartbeat_front_addr":"172.21.15.13:6821/31454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T01:56:10.196 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-17T01:56:10.197 INFO:tasks.ceph:Creating RBD pool 2021-09-17T01:56:10.197 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph --cluster c1 osd pool create rbd 8 2021-09-17T01:56:11.399 INFO:teuthology.orchestra.run.smithi013.stderr:pool 'rbd' created 2021-09-17T01:56:11.412 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-17T01:56:12.403 INFO:teuthology.orchestra.run.smithi013.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-17T01:56:12.418 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-09-17T01:56:12.419 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-17T01:56:12.420 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-17T01:56:12.747 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:12.747 INFO:teuthology.orchestra.run.smithi013.stdout:{"epoch":11,"fsid":"18deeb25-9bb1-48d4-a93a-0bdf7fba8ebb","created":"2021-09-17T01:56:02.057407+0000","modified":"2021-09-17T01:56:12.391424+0000","last_up_change":"2021-09-17T01:56:09.389359+0000","last_in_change":"2021-09-17T01:56:03.032931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-17T01:56:10.527201+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-17T01:56:11.208084+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"09578819-f455-465c-b992-5452e4821af8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6800","nonce":31452},{"type":"v1","addr":"172.21.15.13:6801","nonce":31452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6802","nonce":31452},{"type":"v1","addr":"172.21.15.13:6803","nonce":31452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6806","nonce":31452},{"type":"v1","addr":"172.21.15.13:6807","nonce":31452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6804","nonce":31452},{"type":"v1","addr":"172.21.15.13:6805","nonce":31452}]},"public_addr":"172.21.15.13:6801/31452","cluster_addr":"172.21.15.13:6803/31452","heartbeat_back_addr":"172.21.15.13:6807/31452","heartbeat_front_addr":"172.21.15.13:6805/31452","state":["exists","up"]},{"osd":1,"uuid":"f61e2fdb-384d-4b4a-949c-d035f5b44df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6808","nonce":31453},{"type":"v1","addr":"172.21.15.13:6809","nonce":31453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6810","nonce":31453},{"type":"v1","addr":"172.21.15.13:6811","nonce":31453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6814","nonce":31453},{"type":"v1","addr":"172.21.15.13:6815","nonce":31453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6812","nonce":31453},{"type":"v1","addr":"172.21.15.13:6813","nonce":31453}]},"public_addr":"172.21.15.13:6809/31453","cluster_addr":"172.21.15.13:6811/31453","heartbeat_back_addr":"172.21.15.13:6815/31453","heartbeat_front_addr":"172.21.15.13:6813/31453","state":["exists","up"]},{"osd":2,"uuid":"fe68cba9-61ba-4070-9f71-acc3030ff781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6816","nonce":31454},{"type":"v1","addr":"172.21.15.13:6817","nonce":31454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6818","nonce":31454},{"type":"v1","addr":"172.21.15.13:6819","nonce":31454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6822","nonce":31454},{"type":"v1","addr":"172.21.15.13:6823","nonce":31454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6820","nonce":31454},{"type":"v1","addr":"172.21.15.13:6821","nonce":31454}]},"public_addr":"172.21.15.13:6817/31454","cluster_addr":"172.21.15.13:6819/31454","heartbeat_back_addr":"172.21.15.13:6823/31454","heartbeat_front_addr":"172.21.15.13:6821/31454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:07.398135+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:07.441874+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:07.368197+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T01:56:12.760 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-09-17T01:56:10.527201+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2021-09-17T01:56:11.208084+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2021-09-17T01:56:12.761 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2021-09-17T01:56:13.078 INFO:teuthology.orchestra.run.smithi013.stdout:pg_num: 8 2021-09-17T01:56:13.091 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get .mgr pg_num 2021-09-17T01:56:13.397 INFO:teuthology.orchestra.run.smithi013.stdout:pg_num: 1 2021-09-17T01:56:13.411 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-09-17T01:56:13.412 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-09-17T01:56:13.412 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2021-09-17T01:56:13.760 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:13.772 INFO:teuthology.orchestra.run.smithi013.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6824","nonce":31133},{"type":"v1","addr":"172.21.15.13:6825","nonce":31133}]},"active_addr":"172.21.15.13:6825/31133","active_change":"2021-09-17T01:56:07.160278+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.13:0","nonce":3915314213}]},{"addrvec":[{"type":"v2","addr":"172.21.15.13:0","nonce":748194645}]},{"addrvec":[{"type":"v2","addr":"172.21.15.13:0","nonce":3209720445}]},{"addrvec":[{"type":"v2","addr":"172.21.15.13:0","nonce":1325585127}]}]}} 2021-09-17T01:56:13.774 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-09-17T01:56:13.775 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-09-17T01:56:13.775 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-17T01:56:14.077 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:14.078 INFO:teuthology.orchestra.run.smithi013.stdout:{"epoch":12,"fsid":"18deeb25-9bb1-48d4-a93a-0bdf7fba8ebb","created":"2021-09-17T01:56:02.057407+0000","modified":"2021-09-17T01:56:13.392331+0000","last_up_change":"2021-09-17T01:56:09.389359+0000","last_in_change":"2021-09-17T01:56:03.032931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-17T01:56:10.527201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-17T01:56:11.208084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"09578819-f455-465c-b992-5452e4821af8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6800","nonce":31452},{"type":"v1","addr":"172.21.15.13:6801","nonce":31452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6802","nonce":31452},{"type":"v1","addr":"172.21.15.13:6803","nonce":31452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6806","nonce":31452},{"type":"v1","addr":"172.21.15.13:6807","nonce":31452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6804","nonce":31452},{"type":"v1","addr":"172.21.15.13:6805","nonce":31452}]},"public_addr":"172.21.15.13:6801/31452","cluster_addr":"172.21.15.13:6803/31452","heartbeat_back_addr":"172.21.15.13:6807/31452","heartbeat_front_addr":"172.21.15.13:6805/31452","state":["exists","up"]},{"osd":1,"uuid":"f61e2fdb-384d-4b4a-949c-d035f5b44df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6808","nonce":31453},{"type":"v1","addr":"172.21.15.13:6809","nonce":31453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6810","nonce":31453},{"type":"v1","addr":"172.21.15.13:6811","nonce":31453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6814","nonce":31453},{"type":"v1","addr":"172.21.15.13:6815","nonce":31453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6812","nonce":31453},{"type":"v1","addr":"172.21.15.13:6813","nonce":31453}]},"public_addr":"172.21.15.13:6809/31453","cluster_addr":"172.21.15.13:6811/31453","heartbeat_back_addr":"172.21.15.13:6815/31453","heartbeat_front_addr":"172.21.15.13:6813/31453","state":["exists","up"]},{"osd":2,"uuid":"fe68cba9-61ba-4070-9f71-acc3030ff781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6816","nonce":31454},{"type":"v1","addr":"172.21.15.13:6817","nonce":31454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6818","nonce":31454},{"type":"v1","addr":"172.21.15.13:6819","nonce":31454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6822","nonce":31454},{"type":"v1","addr":"172.21.15.13:6823","nonce":31454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6820","nonce":31454},{"type":"v1","addr":"172.21.15.13:6821","nonce":31454}]},"public_addr":"172.21.15.13:6817/31454","cluster_addr":"172.21.15.13:6819/31454","heartbeat_back_addr":"172.21.15.13:6823/31454","heartbeat_front_addr":"172.21.15.13:6821/31454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:07.398135+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:07.441874+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:07.368197+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T01:56:14.091 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-09-17T01:56:14.091 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-17T01:56:14.389 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:14.390 INFO:teuthology.orchestra.run.smithi013.stdout:{"epoch":12,"fsid":"18deeb25-9bb1-48d4-a93a-0bdf7fba8ebb","created":"2021-09-17T01:56:02.057407+0000","modified":"2021-09-17T01:56:13.392331+0000","last_up_change":"2021-09-17T01:56:09.389359+0000","last_in_change":"2021-09-17T01:56:03.032931+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-17T01:56:10.527201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-17T01:56:11.208084+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"09578819-f455-465c-b992-5452e4821af8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6800","nonce":31452},{"type":"v1","addr":"172.21.15.13:6801","nonce":31452}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6802","nonce":31452},{"type":"v1","addr":"172.21.15.13:6803","nonce":31452}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6806","nonce":31452},{"type":"v1","addr":"172.21.15.13:6807","nonce":31452}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6804","nonce":31452},{"type":"v1","addr":"172.21.15.13:6805","nonce":31452}]},"public_addr":"172.21.15.13:6801/31452","cluster_addr":"172.21.15.13:6803/31452","heartbeat_back_addr":"172.21.15.13:6807/31452","heartbeat_front_addr":"172.21.15.13:6805/31452","state":["exists","up"]},{"osd":1,"uuid":"f61e2fdb-384d-4b4a-949c-d035f5b44df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6808","nonce":31453},{"type":"v1","addr":"172.21.15.13:6809","nonce":31453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6810","nonce":31453},{"type":"v1","addr":"172.21.15.13:6811","nonce":31453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6814","nonce":31453},{"type":"v1","addr":"172.21.15.13:6815","nonce":31453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6812","nonce":31453},{"type":"v1","addr":"172.21.15.13:6813","nonce":31453}]},"public_addr":"172.21.15.13:6809/31453","cluster_addr":"172.21.15.13:6811/31453","heartbeat_back_addr":"172.21.15.13:6815/31453","heartbeat_front_addr":"172.21.15.13:6813/31453","state":["exists","up"]},{"osd":2,"uuid":"fe68cba9-61ba-4070-9f71-acc3030ff781","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6816","nonce":31454},{"type":"v1","addr":"172.21.15.13:6817","nonce":31454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6818","nonce":31454},{"type":"v1","addr":"172.21.15.13:6819","nonce":31454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6822","nonce":31454},{"type":"v1","addr":"172.21.15.13:6823","nonce":31454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6820","nonce":31454},{"type":"v1","addr":"172.21.15.13:6821","nonce":31454}]},"public_addr":"172.21.15.13:6817/31454","cluster_addr":"172.21.15.13:6819/31454","heartbeat_back_addr":"172.21.15.13:6823/31454","heartbeat_front_addr":"172.21.15.13:6821/31454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:07.398135+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:07.441874+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:07.368197+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T01:56:14.403 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2021-09-17T01:56:14.553 INFO:teuthology.orchestra.run.smithi013.stdout:34359738371 2021-09-17T01:56:14.553 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2021-09-17T01:56:14.703 INFO:teuthology.orchestra.run.smithi013.stdout:34359738371 2021-09-17T01:56:14.704 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2021-09-17T01:56:14.853 INFO:teuthology.orchestra.run.smithi013.stdout:34359738371 2021-09-17T01:56:14.854 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-09-17T01:56:15.157 INFO:teuthology.orchestra.run.smithi013.stdout:34359738370 2021-09-17T01:56:15.171 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2021-09-17T01:56:16.172 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-09-17T01:56:16.490 INFO:teuthology.orchestra.run.smithi013.stdout:34359738371 2021-09-17T01:56:16.503 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2021-09-17T01:56:16.504 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-09-17T01:56:16.809 INFO:teuthology.orchestra.run.smithi013.stdout:34359738371 2021-09-17T01:56:16.823 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2021-09-17T01:56:16.823 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2021-09-17T01:56:17.128 INFO:teuthology.orchestra.run.smithi013.stdout:34359738371 2021-09-17T01:56:17.141 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.2 2021-09-17T01:56:17.142 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-09-17T01:56:17.142 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-17T01:56:17.441 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:17.443 INFO:teuthology.orchestra.run.smithi013.stderr:dumped all 2021-09-17T01:56:17.454 INFO:teuthology.orchestra.run.smithi013.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2021-09-17T01:56:17.165360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24916,"kb_used_data":1468,"kb_used_omap":0,"kb_used_meta":23168,"kb_avail":314547884,"statfs":{"total":322122547200,"available":322097033216,"internally_reserved":0,"allocated":1503232,"data_stored":1265542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23724032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000375"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.400946+0000","last_change":"2021-09-17T01:56:12.399474+0000","last_active":"2021-09-17T01:56:12.400946+0000","last_peered":"2021-09-17T01:56:12.400946+0000","last_clean":"2021-09-17T01:56:12.400946+0000","last_became_active":"2021-09-17T01:56:12.399224+0000","last_became_peered":"2021-09-17T01:56:12.399224+0000","last_unstale":"2021-09-17T01:56:12.400946+0000","last_undegraded":"2021-09-17T01:56:12.400946+0000","last_fullsized":"2021-09-17T01:56:12.400946+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.400807+0000","last_change":"2021-09-17T01:56:12.399503+0000","last_active":"2021-09-17T01:56:12.400807+0000","last_peered":"2021-09-17T01:56:12.400807+0000","last_clean":"2021-09-17T01:56:12.400807+0000","last_became_active":"2021-09-17T01:56:12.399186+0000","last_became_peered":"2021-09-17T01:56:12.399186+0000","last_unstale":"2021-09-17T01:56:12.400807+0000","last_undegraded":"2021-09-17T01:56:12.400807+0000","last_fullsized":"2021-09-17T01:56:12.400807+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.401608+0000","last_change":"2021-09-17T01:56:12.399969+0000","last_active":"2021-09-17T01:56:12.401608+0000","last_peered":"2021-09-17T01:56:12.401608+0000","last_clean":"2021-09-17T01:56:12.401608+0000","last_became_active":"2021-09-17T01:56:12.399696+0000","last_became_peered":"2021-09-17T01:56:12.399696+0000","last_unstale":"2021-09-17T01:56:12.401608+0000","last_undegraded":"2021-09-17T01:56:12.401608+0000","last_fullsized":"2021-09-17T01:56:12.401608+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.400277+0000","last_change":"2021-09-17T01:56:12.398870+0000","last_active":"2021-09-17T01:56:12.400277+0000","last_peered":"2021-09-17T01:56:12.400277+0000","last_clean":"2021-09-17T01:56:12.400277+0000","last_became_active":"2021-09-17T01:56:12.398586+0000","last_became_peered":"2021-09-17T01:56:12.398586+0000","last_unstale":"2021-09-17T01:56:12.400277+0000","last_undegraded":"2021-09-17T01:56:12.400277+0000","last_fullsized":"2021-09-17T01:56:12.400277+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.402043+0000","last_change":"2021-09-17T01:56:12.401657+0000","last_active":"2021-09-17T01:56:12.402043+0000","last_peered":"2021-09-17T01:56:12.402043+0000","last_clean":"2021-09-17T01:56:12.402043+0000","last_became_active":"2021-09-17T01:56:12.401372+0000","last_became_peered":"2021-09-17T01:56:12.401372+0000","last_unstale":"2021-09-17T01:56:12.402043+0000","last_undegraded":"2021-09-17T01:56:12.402043+0000","last_fullsized":"2021-09-17T01:56:12.402043+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.901007+0000","last_change":"2021-09-17T01:56:12.401134+0000","last_active":"2021-09-17T01:56:12.901007+0000","last_peered":"2021-09-17T01:56:12.901007+0000","last_clean":"2021-09-17T01:56:12.901007+0000","last_became_active":"2021-09-17T01:56:12.400821+0000","last_became_peered":"2021-09-17T01:56:12.400821+0000","last_unstale":"2021-09-17T01:56:12.901007+0000","last_undegraded":"2021-09-17T01:56:12.901007+0000","last_fullsized":"2021-09-17T01:56:12.901007+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.400360+0000","last_change":"2021-09-17T01:56:12.398930+0000","last_active":"2021-09-17T01:56:12.400360+0000","last_peered":"2021-09-17T01:56:12.400360+0000","last_clean":"2021-09-17T01:56:12.400360+0000","last_became_active":"2021-09-17T01:56:12.398655+0000","last_became_peered":"2021-09-17T01:56:12.398655+0000","last_unstale":"2021-09-17T01:56:12.400360+0000","last_undegraded":"2021-09-17T01:56:12.400360+0000","last_fullsized":"2021-09-17T01:56:12.400360+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.400327+0000","last_change":"2021-09-17T01:56:12.398850+0000","last_active":"2021-09-17T01:56:12.400327+0000","last_peered":"2021-09-17T01:56:12.400327+0000","last_clean":"2021-09-17T01:56:12.400327+0000","last_became_active":"2021-09-17T01:56:12.398549+0000","last_became_peered":"2021-09-17T01:56:12.398549+0000","last_unstale":"2021-09-17T01:56:12.400327+0000","last_undegraded":"2021-09-17T01:56:12.400327+0000","last_fullsized":"2021-09-17T01:56:12.400327+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.399199+0000","last_change":"2021-09-17T01:56:12.398764+0000","last_active":"2021-09-17T01:56:12.399199+0000","last_peered":"2021-09-17T01:56:12.399199+0000","last_clean":"2021-09-17T01:56:12.399199+0000","last_became_active":"2021-09-17T01:56:12.398539+0000","last_became_peered":"2021-09-17T01:56:12.398539+0000","last_unstale":"2021-09-17T01:56:12.399199+0000","last_undegraded":"2021-09-17T01:56:12.399199+0000","last_fullsized":"2021-09-17T01:56:12.399199+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8628,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104848972,"statfs":{"total":107374182400,"available":107365347328,"internally_reserved":0,"allocated":700416,"data_stored":618949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7600,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850000,"statfs":{"total":107374182400,"available":107366400000,"internally_reserved":0,"allocated":106496,"data_stored":28581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8688,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848912,"statfs":{"total":107374182400,"available":107365285888,"internally_reserved":0,"allocated":696320,"data_stored":618012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-17T01:56:17.455 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-17T01:56:17.755 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:17.756 INFO:teuthology.orchestra.run.smithi013.stderr:dumped all 2021-09-17T01:56:17.767 INFO:teuthology.orchestra.run.smithi013.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2021-09-17T01:56:17.165360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24916,"kb_used_data":1468,"kb_used_omap":0,"kb_used_meta":23168,"kb_avail":314547884,"statfs":{"total":322122547200,"available":322097033216,"internally_reserved":0,"allocated":1503232,"data_stored":1265542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23724032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000375"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.400946+0000","last_change":"2021-09-17T01:56:12.399474+0000","last_active":"2021-09-17T01:56:12.400946+0000","last_peered":"2021-09-17T01:56:12.400946+0000","last_clean":"2021-09-17T01:56:12.400946+0000","last_became_active":"2021-09-17T01:56:12.399224+0000","last_became_peered":"2021-09-17T01:56:12.399224+0000","last_unstale":"2021-09-17T01:56:12.400946+0000","last_undegraded":"2021-09-17T01:56:12.400946+0000","last_fullsized":"2021-09-17T01:56:12.400946+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.400807+0000","last_change":"2021-09-17T01:56:12.399503+0000","last_active":"2021-09-17T01:56:12.400807+0000","last_peered":"2021-09-17T01:56:12.400807+0000","last_clean":"2021-09-17T01:56:12.400807+0000","last_became_active":"2021-09-17T01:56:12.399186+0000","last_became_peered":"2021-09-17T01:56:12.399186+0000","last_unstale":"2021-09-17T01:56:12.400807+0000","last_undegraded":"2021-09-17T01:56:12.400807+0000","last_fullsized":"2021-09-17T01:56:12.400807+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.401608+0000","last_change":"2021-09-17T01:56:12.399969+0000","last_active":"2021-09-17T01:56:12.401608+0000","last_peered":"2021-09-17T01:56:12.401608+0000","last_clean":"2021-09-17T01:56:12.401608+0000","last_became_active":"2021-09-17T01:56:12.399696+0000","last_became_peered":"2021-09-17T01:56:12.399696+0000","last_unstale":"2021-09-17T01:56:12.401608+0000","last_undegraded":"2021-09-17T01:56:12.401608+0000","last_fullsized":"2021-09-17T01:56:12.401608+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.400277+0000","last_change":"2021-09-17T01:56:12.398870+0000","last_active":"2021-09-17T01:56:12.400277+0000","last_peered":"2021-09-17T01:56:12.400277+0000","last_clean":"2021-09-17T01:56:12.400277+0000","last_became_active":"2021-09-17T01:56:12.398586+0000","last_became_peered":"2021-09-17T01:56:12.398586+0000","last_unstale":"2021-09-17T01:56:12.400277+0000","last_undegraded":"2021-09-17T01:56:12.400277+0000","last_fullsized":"2021-09-17T01:56:12.400277+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.402043+0000","last_change":"2021-09-17T01:56:12.401657+0000","last_active":"2021-09-17T01:56:12.402043+0000","last_peered":"2021-09-17T01:56:12.402043+0000","last_clean":"2021-09-17T01:56:12.402043+0000","last_became_active":"2021-09-17T01:56:12.401372+0000","last_became_peered":"2021-09-17T01:56:12.401372+0000","last_unstale":"2021-09-17T01:56:12.402043+0000","last_undegraded":"2021-09-17T01:56:12.402043+0000","last_fullsized":"2021-09-17T01:56:12.402043+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.901007+0000","last_change":"2021-09-17T01:56:12.401134+0000","last_active":"2021-09-17T01:56:12.901007+0000","last_peered":"2021-09-17T01:56:12.901007+0000","last_clean":"2021-09-17T01:56:12.901007+0000","last_became_active":"2021-09-17T01:56:12.400821+0000","last_became_peered":"2021-09-17T01:56:12.400821+0000","last_unstale":"2021-09-17T01:56:12.901007+0000","last_undegraded":"2021-09-17T01:56:12.901007+0000","last_fullsized":"2021-09-17T01:56:12.901007+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.400360+0000","last_change":"2021-09-17T01:56:12.398930+0000","last_active":"2021-09-17T01:56:12.400360+0000","last_peered":"2021-09-17T01:56:12.400360+0000","last_clean":"2021-09-17T01:56:12.400360+0000","last_became_active":"2021-09-17T01:56:12.398655+0000","last_became_peered":"2021-09-17T01:56:12.398655+0000","last_unstale":"2021-09-17T01:56:12.400360+0000","last_undegraded":"2021-09-17T01:56:12.400360+0000","last_fullsized":"2021-09-17T01:56:12.400360+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.400327+0000","last_change":"2021-09-17T01:56:12.398850+0000","last_active":"2021-09-17T01:56:12.400327+0000","last_peered":"2021-09-17T01:56:12.400327+0000","last_clean":"2021-09-17T01:56:12.400327+0000","last_became_active":"2021-09-17T01:56:12.398549+0000","last_became_peered":"2021-09-17T01:56:12.398549+0000","last_unstale":"2021-09-17T01:56:12.400327+0000","last_undegraded":"2021-09-17T01:56:12.400327+0000","last_fullsized":"2021-09-17T01:56:12.400327+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:12.399199+0000","last_change":"2021-09-17T01:56:12.398764+0000","last_active":"2021-09-17T01:56:12.399199+0000","last_peered":"2021-09-17T01:56:12.399199+0000","last_clean":"2021-09-17T01:56:12.399199+0000","last_became_active":"2021-09-17T01:56:12.398539+0000","last_became_peered":"2021-09-17T01:56:12.398539+0000","last_unstale":"2021-09-17T01:56:12.399199+0000","last_undegraded":"2021-09-17T01:56:12.399199+0000","last_fullsized":"2021-09-17T01:56:12.399199+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:11.390576+0000","last_clean_scrub_stamp":"2021-09-17T01:56:11.390576+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8628,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104848972,"statfs":{"total":107374182400,"available":107365347328,"internally_reserved":0,"allocated":700416,"data_stored":618949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7600,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850000,"statfs":{"total":107374182400,"available":107366400000,"internally_reserved":0,"allocated":106496,"data_stored":28581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8688,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848912,"statfs":{"total":107374182400,"available":107365285888,"internally_reserved":0,"allocated":696320,"data_stored":618012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-17T01:56:17.768 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-09-17T01:56:17.769 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-09-17T01:56:17.769 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-09-17T01:56:17.769 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2021-09-17T01:56:18.136 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:18.137 INFO:teuthology.orchestra.run.smithi013.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-17T01:56:18.150 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-09-17T01:56:18.150 INFO:teuthology.run_tasks:Running task ceph... 2021-09-17T01:56:18.160 INFO:tasks.ceph:Creating ceph cluster c2... 2021-09-17T01:56:18.161 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-17T01:56:18.161 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/ivancich-2021-09-16_17:11:22-rgw-wip-eric-testing-2-distro-basic-smithi/6392884', 'branch': 'wip-eric-testing-2', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': 'ivancich@redhat.com', 'first_in_suite': False, 'job_id': '6392884', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'ivancich-2021-09-16_17:11:22-rgw-wip-eric-testing-2-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-eric-testing-2'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-eric-testing-2', 'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049'}}, 'owner': 'scheduled_ivancich@teuthology', 'priority': 83, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-eric-testing-2', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_2a911e49300877ce783c2becf9f484a85fa1c049/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '2a911e49300877ce783c2becf9f484a85fa1c049', 'targets': {'smithi013.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQUSlRIEp7IBLYORccR8lUdTmBJ7wmpIUUDFB3BhcT3ksU2F+8QkMCOspCnZF1sYWLW7px9JhbwBIFpTNh82FaIYm+10eqZjBj0nwoTruKjA2DxTEat/JJX2bMF71TbOrjDQwZccmz8TZGYhIlK2tHMoOPB8rGEcM3AHllSFf64/M7lUiZlIZOOEChRvepzkPTfdyHE7lBCH+PJ9pCabnK17OjVgmBGEkUWbbprw6HadBeQ1CrgVzwjabqyAd/8DCvTY2OxE/k+sVVZe4U6kEdBcRe9erCVlQxjoXlPchpL+k0hA2+rk8LHkurz/mdT9av4QVYxi8YcTKIfjYbwhgq5eu964h+rl3y4oZ/NIk+X8MOzsHViYYfHA5WXCRDcZxqUojCCL5+VntRfbe/cwNK9UGdV4LJ3F9dNSw9tYEAvtzE7Mh7xzHOPntmtg/pL43RboxM/YqQEp5wAGNFLPBjhhfsx+54JpWVZBYw5hjU9m36xlETxeS60ZMpA9ZpiuE=', 'smithi045.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCulDN10/Vke3BN23zcEbwKxvhTR910c5AgG82/Ohf4WiNPYA98666PzF0PZudRGkkX89oUG+zPhqFxQc+j3oGHtRyORPLZzJPG//ZKLtwr1WONI012azN1JgotupRG+CwfHM7sThE09XdXiYLDYsIj9R5NDPT8ATTKxRAMVpMUuPx0MjyS9R7c4mVQ+ZKEDq8hRS1hl5svH+iBtEooVtLGZljHsj8HvHDjdtrrTxbSkzhZxHlnD6pEm2SngtbAglAiMO5NuYltRDf8x/k6V3eBODOz4rJUdDtsif9nCjU2HUEcdnNTyZlRJizeshvySk67OoZ5fc2eezTmluQ6QIcv+B0dLUG3Eor1oqHCdtxzBR/PJMZo9tLppdtH9AHitodCgOkdMJyd/zzLBF9iDLBsrXPQ20wVwb5rqnqzRZIpz28zreE9wrS07Ny5FfkexvGofoqbbDpjm8sshrtwv/jfcPcTYd0NUhL49SuSyhOWFGpYKvx8pgAmEwz2EFTvfVc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2a911e49300877ce783c2becf9f484a85fa1c049', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '63133ca92c299bd23f40f94d370cf5381b0b1313', 'timestamp': '2021-09-16_17:11:22', 'tube': 'smithi', 'user': 'ivancich', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-17T01:56:18.162 DEBUG:teuthology.orchestra.run.smithi013:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-17T01:56:18.164 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-17T01:56:18.183 DEBUG:teuthology.orchestra.run.smithi013:> sudo install -d -m0777 -- /var/run/ceph 2021-09-17T01:56:18.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/run/ceph 2021-09-17T01:56:18.256 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:18.257 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2021-09-17T01:56:18.314 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-17T01:56:18.314 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-17T01:56:18.370 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-17T01:56:18.371 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T01:56:18.371 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 113104 Links: 1 2021-09-17T01:56:18.371 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T01:56:18.372 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2021-09-17T01:56:18.372 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2021-09-17 01:55:06.739725989 +0000 2021-09-17T01:56:18.372 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2021-09-17 01:45:30.412516525 +0000 2021-09-17T01:56:18.372 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2021-09-17 01:45:30.412516525 +0000 2021-09-17T01:56:18.372 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2021-09-17T01:56:18.382 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2021-09-17T01:56:18.382 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2021-09-17T01:56:18.383 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000116799 s, 4.4 MB/s 2021-09-17T01:56:18.385 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-17T01:56:18.442 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-17T01:56:18.443 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T01:56:18.443 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 111592 Links: 1 2021-09-17T01:56:18.443 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T01:56:18.443 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2021-09-17T01:56:18.444 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2021-09-17 01:55:06.739725989 +0000 2021-09-17T01:56:18.444 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2021-09-17 01:45:30.789510709 +0000 2021-09-17T01:56:18.444 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2021-09-17 01:45:30.789510709 +0000 2021-09-17T01:56:18.444 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2021-09-17T01:56:18.454 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2021-09-17T01:56:18.454 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2021-09-17T01:56:18.454 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000112843 s, 4.5 MB/s 2021-09-17T01:56:18.457 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-17T01:56:18.514 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-17T01:56:18.514 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T01:56:18.515 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 118818 Links: 1 2021-09-17T01:56:18.515 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T01:56:18.515 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2021-09-17T01:56:18.516 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2021-09-17 01:55:06.739725989 +0000 2021-09-17T01:56:18.516 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2021-09-17 01:45:31.131505432 +0000 2021-09-17T01:56:18.516 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2021-09-17 01:45:31.131505432 +0000 2021-09-17T01:56:18.517 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2021-09-17T01:56:18.524 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2021-09-17T01:56:18.525 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2021-09-17T01:56:18.525 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000103727 s, 4.9 MB/s 2021-09-17T01:56:18.528 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-17T01:56:18.583 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-17T01:56:18.584 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T01:56:18.584 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 6h/6d Inode: 118926 Links: 1 2021-09-17T01:56:18.584 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T01:56:18.585 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2021-09-17T01:56:18.585 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2021-09-17 01:55:06.739725989 +0000 2021-09-17T01:56:18.585 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2021-09-17 01:45:31.479500063 +0000 2021-09-17T01:56:18.585 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2021-09-17 01:45:31.479500063 +0000 2021-09-17T01:56:18.585 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2021-09-17T01:56:18.594 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2021-09-17T01:56:18.595 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2021-09-17T01:56:18.595 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000102525 s, 5.0 MB/s 2021-09-17T01:56:18.596 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T01:56:18.597 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-17T01:56:18.597 INFO:tasks.ceph:Generating config... 2021-09-17T01:56:18.599 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-17T01:56:18.599 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-17T01:56:18.600 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-17T01:56:18.600 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T01:56:18.600 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-17T01:56:18.601 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-17T01:56:18.601 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-17T01:56:18.601 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-17T01:56:18.602 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-17T01:56:18.602 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-17T01:56:18.602 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-17T01:56:18.602 INFO:tasks.ceph:[client] setuser = ceph 2021-09-17T01:56:18.603 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-17T01:56:18.603 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-17T01:56:18.603 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-17T01:56:18.604 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-17T01:56:18.604 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-17T01:56:18.604 INFO:tasks.ceph:[osd] debug ms = 1 2021-09-17T01:56:18.605 INFO:tasks.ceph:[osd] debug osd = 20 2021-09-17T01:56:18.605 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-17T01:56:18.605 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-17T01:56:18.605 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-17T01:56:18.606 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-17T01:56:18.606 INFO:tasks.ceph:Setting up c2.mon.a... 2021-09-17T01:56:18.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-09-17T01:56:18.690 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/c2.keyring 2021-09-17T01:56:18.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-09-17T01:56:18.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-17T01:56:18.797 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.45')] 2021-09-17T01:56:18.798 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.45'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-17T01:56:18.799 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:18.800 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-17T01:56:18.852 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.45 --print /home/ubuntu/cephtest/c2.monmap 2021-09-17T01:56:18.927 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:56:18.928 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:56:18.929 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-09-17T01:56:18.929 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: generated fsid e009c937-b2b5-4685-bce2-3ad003fa6b2b 2021-09-17T01:56:18.929 INFO:teuthology.orchestra.run.smithi045.stdout:setting min_mon_release = octopus 2021-09-17T01:56:18.930 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 0 2021-09-17T01:56:18.930 INFO:teuthology.orchestra.run.smithi045.stdout:fsid e009c937-b2b5-4685-bce2-3ad003fa6b2b 2021-09-17T01:56:18.930 INFO:teuthology.orchestra.run.smithi045.stdout:last_changed 2021-09-17T01:56:18.928007+0000 2021-09-17T01:56:18.930 INFO:teuthology.orchestra.run.smithi045.stdout:created 2021-09-17T01:56:18.928007+0000 2021-09-17T01:56:18.930 INFO:teuthology.orchestra.run.smithi045.stdout:min_mon_release 15 (octopus) 2021-09-17T01:56:18.931 INFO:teuthology.orchestra.run.smithi045.stdout:election_strategy: 1 2021-09-17T01:56:18.931 INFO:teuthology.orchestra.run.smithi045.stdout:0: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.a 2021-09-17T01:56:18.931 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-09-17T01:56:18.932 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-17T01:56:18.984 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID e009c937-b2b5-4685-bce2-3ad003fa6b2b... 2021-09-17T01:56:18.985 DEBUG:teuthology.orchestra.run.smithi013:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-17T01:56:18.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-17T01:56:19.037 INFO:teuthology.orchestra.run.smithi013.stdout:[global] 2021-09-17T01:56:19.038 INFO:teuthology.orchestra.run.smithi013.stdout: chdir = "" 2021-09-17T01:56:19.038 INFO:teuthology.orchestra.run.smithi013.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-17T01:56:19.038 INFO:teuthology.orchestra.run.smithi013.stdout: auth supported = cephx 2021-09-17T01:56:19.039 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.039 INFO:teuthology.orchestra.run.smithi013.stdout: filestore xattr use omap = true 2021-09-17T01:56:19.039 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.039 INFO:teuthology.orchestra.run.smithi013.stdout: mon clock drift allowed = 1.000 2021-09-17T01:56:19.039 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.040 INFO:teuthology.orchestra.run.smithi013.stdout: osd crush chooseleaf type = 0 2021-09-17T01:56:19.040 INFO:teuthology.orchestra.run.smithi013.stdout: auth debug = true 2021-09-17T01:56:19.040 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.040 INFO:teuthology.orchestra.run.smithi013.stdout: ms die on old message = true 2021-09-17T01:56:19.041 INFO:teuthology.orchestra.run.smithi013.stdout: ms die on bug = true 2021-09-17T01:56:19.041 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.041 INFO:teuthology.orchestra.run.smithi013.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-17T01:56:19.041 INFO:teuthology.orchestra.run.smithi013.stdout: mon pg warn max object skew = 0 2021-09-17T01:56:19.041 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.042 INFO:teuthology.orchestra.run.smithi013.stdout: # disable pg_autoscaler by default for new pools 2021-09-17T01:56:19.042 INFO:teuthology.orchestra.run.smithi013.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-17T01:56:19.042 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.042 INFO:teuthology.orchestra.run.smithi013.stdout: osd pool default size = 2 2021-09-17T01:56:19.042 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.043 INFO:teuthology.orchestra.run.smithi013.stdout: mon osd allow primary affinity = true 2021-09-17T01:56:19.043 INFO:teuthology.orchestra.run.smithi013.stdout: mon osd allow pg remap = true 2021-09-17T01:56:19.043 INFO:teuthology.orchestra.run.smithi013.stdout: mon warn on legacy crush tunables = false 2021-09-17T01:56:19.043 INFO:teuthology.orchestra.run.smithi013.stdout: mon warn on crush straw calc version zero = false 2021-09-17T01:56:19.043 INFO:teuthology.orchestra.run.smithi013.stdout: mon warn on no sortbitwise = false 2021-09-17T01:56:19.044 INFO:teuthology.orchestra.run.smithi013.stdout: mon warn on osd down out interval zero = false 2021-09-17T01:56:19.044 INFO:teuthology.orchestra.run.smithi013.stdout: mon warn on too few osds = false 2021-09-17T01:56:19.044 INFO:teuthology.orchestra.run.smithi013.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-17T01:56:19.044 INFO:teuthology.orchestra.run.smithi013.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-17T01:56:19.044 INFO:teuthology.orchestra.run.smithi013.stdout: mon_allow_pool_size_one = true 2021-09-17T01:56:19.045 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.045 INFO:teuthology.orchestra.run.smithi013.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-17T01:56:19.045 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.045 INFO:teuthology.orchestra.run.smithi013.stdout: osd default data pool replay window = 5 2021-09-17T01:56:19.045 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.046 INFO:teuthology.orchestra.run.smithi013.stdout: mon allow pool delete = true 2021-09-17T01:56:19.046 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.046 INFO:teuthology.orchestra.run.smithi013.stdout: mon cluster log file level = debug 2021-09-17T01:56:19.046 INFO:teuthology.orchestra.run.smithi013.stdout: debug asserts on shutdown = true 2021-09-17T01:56:19.046 INFO:teuthology.orchestra.run.smithi013.stdout: mon health detail to clog = false 2021-09-17T01:56:19.047 INFO:teuthology.orchestra.run.smithi013.stdout: mon host = 172.21.15.45 2021-09-17T01:56:19.047 INFO:teuthology.orchestra.run.smithi013.stdout: fsid = e009c937-b2b5-4685-bce2-3ad003fa6b2b 2021-09-17T01:56:19.047 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.047 INFO:teuthology.orchestra.run.smithi013.stdout:[osd] 2021-09-17T01:56:19.047 INFO:teuthology.orchestra.run.smithi013.stdout: osd journal size = 100 2021-09-17T01:56:19.048 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.048 INFO:teuthology.orchestra.run.smithi013.stdout: osd scrub load threshold = 5.0 2021-09-17T01:56:19.048 INFO:teuthology.orchestra.run.smithi013.stdout: osd scrub max interval = 600 2021-09-17T01:56:19.048 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.048 INFO:teuthology.orchestra.run.smithi013.stdout: osd recover clone overlap = true 2021-09-17T01:56:19.049 INFO:teuthology.orchestra.run.smithi013.stdout: osd recovery max chunk = 1048576 2021-09-17T01:56:19.049 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.049 INFO:teuthology.orchestra.run.smithi013.stdout: osd debug shutdown = true 2021-09-17T01:56:19.049 INFO:teuthology.orchestra.run.smithi013.stdout: osd debug op order = true 2021-09-17T01:56:19.049 INFO:teuthology.orchestra.run.smithi013.stdout: osd debug verify stray on activate = true 2021-09-17T01:56:19.050 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.050 INFO:teuthology.orchestra.run.smithi013.stdout: osd open classes on start = true 2021-09-17T01:56:19.050 INFO:teuthology.orchestra.run.smithi013.stdout: osd debug pg log writeout = true 2021-09-17T01:56:19.050 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.050 INFO:teuthology.orchestra.run.smithi013.stdout: osd deep scrub update digest min age = 30 2021-09-17T01:56:19.051 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.051 INFO:teuthology.orchestra.run.smithi013.stdout: osd map max advance = 10 2021-09-17T01:56:19.051 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.051 INFO:teuthology.orchestra.run.smithi013.stdout: journal zero on create = true 2021-09-17T01:56:19.051 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.052 INFO:teuthology.orchestra.run.smithi013.stdout: filestore ondisk finisher threads = 3 2021-09-17T01:56:19.052 INFO:teuthology.orchestra.run.smithi013.stdout: filestore apply finisher threads = 3 2021-09-17T01:56:19.052 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.052 INFO:teuthology.orchestra.run.smithi013.stdout: bdev debug aio = true 2021-09-17T01:56:19.052 INFO:teuthology.orchestra.run.smithi013.stdout: osd debug misdirected ops = true 2021-09-17T01:56:19.053 INFO:teuthology.orchestra.run.smithi013.stdout: debug ms = 1 2021-09-17T01:56:19.053 INFO:teuthology.orchestra.run.smithi013.stdout: debug osd = 20 2021-09-17T01:56:19.053 INFO:teuthology.orchestra.run.smithi013.stdout: rocksdb delete range threshold = 0 2021-09-17T01:56:19.053 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.053 INFO:teuthology.orchestra.run.smithi013.stdout:[mgr] 2021-09-17T01:56:19.054 INFO:teuthology.orchestra.run.smithi013.stdout: debug ms = 1 2021-09-17T01:56:19.054 INFO:teuthology.orchestra.run.smithi013.stdout: debug mgr = 20 2021-09-17T01:56:19.054 INFO:teuthology.orchestra.run.smithi013.stdout: debug mon = 20 2021-09-17T01:56:19.054 INFO:teuthology.orchestra.run.smithi013.stdout: debug auth = 20 2021-09-17T01:56:19.055 INFO:teuthology.orchestra.run.smithi013.stdout: mon reweight min pgs per osd = 4 2021-09-17T01:56:19.055 INFO:teuthology.orchestra.run.smithi013.stdout: mon reweight min bytes per osd = 10 2021-09-17T01:56:19.055 INFO:teuthology.orchestra.run.smithi013.stdout: mgr/telemetry/nag = false 2021-09-17T01:56:19.055 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.055 INFO:teuthology.orchestra.run.smithi013.stdout:[mon] 2021-09-17T01:56:19.056 INFO:teuthology.orchestra.run.smithi013.stdout: debug ms = 1 2021-09-17T01:56:19.056 INFO:teuthology.orchestra.run.smithi013.stdout: debug mon = 20 2021-09-17T01:56:19.056 INFO:teuthology.orchestra.run.smithi013.stdout: debug paxos = 20 2021-09-17T01:56:19.056 INFO:teuthology.orchestra.run.smithi013.stdout: debug auth = 20 2021-09-17T01:56:19.056 INFO:teuthology.orchestra.run.smithi013.stdout: mon data avail warn = 5 2021-09-17T01:56:19.056 INFO:teuthology.orchestra.run.smithi013.stdout: mon mgr mkfs grace = 240 2021-09-17T01:56:19.057 INFO:teuthology.orchestra.run.smithi013.stdout: mon reweight min pgs per osd = 4 2021-09-17T01:56:19.057 INFO:teuthology.orchestra.run.smithi013.stdout: mon osd reporter subtree level = osd 2021-09-17T01:56:19.057 INFO:teuthology.orchestra.run.smithi013.stdout: mon osd prime pg temp = true 2021-09-17T01:56:19.057 INFO:teuthology.orchestra.run.smithi013.stdout: mon reweight min bytes per osd = 10 2021-09-17T01:56:19.058 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.058 INFO:teuthology.orchestra.run.smithi013.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-17T01:56:19.058 INFO:teuthology.orchestra.run.smithi013.stdout: auth mon ticket ttl = 660 # 11m 2021-09-17T01:56:19.058 INFO:teuthology.orchestra.run.smithi013.stdout: auth service ticket ttl = 240 # 4m 2021-09-17T01:56:19.058 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.059 INFO:teuthology.orchestra.run.smithi013.stdout: # don't complain about insecure global_id in the test suite 2021-09-17T01:56:19.059 INFO:teuthology.orchestra.run.smithi013.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-17T01:56:19.059 INFO:teuthology.orchestra.run.smithi013.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-17T01:56:19.059 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:56:19.059 INFO:teuthology.orchestra.run.smithi013.stdout:[client] 2021-09-17T01:56:19.059 INFO:teuthology.orchestra.run.smithi013.stdout: rgw cache enabled = true 2021-09-17T01:56:19.060 INFO:teuthology.orchestra.run.smithi013.stdout: rgw enable ops log = true 2021-09-17T01:56:19.060 INFO:teuthology.orchestra.run.smithi013.stdout: rgw enable usage log = true 2021-09-17T01:56:19.060 INFO:teuthology.orchestra.run.smithi013.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-17T01:56:19.060 INFO:teuthology.orchestra.run.smithi013.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-17T01:56:19.060 INFO:teuthology.orchestra.run.smithi013.stdout: debug rgw = 20 2021-09-17T01:56:19.061 INFO:teuthology.orchestra.run.smithi013.stdout: rgw crypt require ssl = False 2021-09-17T01:56:19.061 INFO:teuthology.orchestra.run.smithi013.stdout: rgw crypt s3 kms backend = testing 2021-09-17T01:56:19.061 INFO:teuthology.orchestra.run.smithi013.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T01:56:19.061 INFO:teuthology.orchestra.run.smithi013.stdout: rgw curl low speed time = 300 2021-09-17T01:56:19.061 INFO:teuthology.orchestra.run.smithi013.stdout: rgw data log num shards = 4 2021-09-17T01:56:19.062 INFO:teuthology.orchestra.run.smithi013.stdout: rgw md log max shards = 4 2021-09-17T01:56:19.062 INFO:teuthology.orchestra.run.smithi013.stdout: rgw sync log trim interval = 0 2021-09-17T01:56:19.062 INFO:teuthology.orchestra.run.smithi013.stdout: rgw sync meta inject err probability = 0.1 2021-09-17T01:56:19.062 INFO:teuthology.orchestra.run.smithi013.stdout: rgw sync obj etag verify = True 2021-09-17T01:56:19.062 INFO:teuthology.orchestra.run.smithi013.stdout: setgroup = ceph 2021-09-17T01:56:19.063 INFO:teuthology.orchestra.run.smithi013.stdout: setuser = ceph 2021-09-17T01:56:19.063 INFO:teuthology.orchestra.run.smithi013.stdout:[mon.a] 2021-09-17T01:56:19.063 INFO:teuthology.orchestra.run.smithi013.stdout:[osd.0] 2021-09-17T01:56:19.063 INFO:teuthology.orchestra.run.smithi013.stdout: osd_max_omap_entries_per_request = 10 2021-09-17T01:56:19.063 INFO:teuthology.orchestra.run.smithi013.stdout:[osd.1] 2021-09-17T01:56:19.064 INFO:teuthology.orchestra.run.smithi013.stdout: osd_max_omap_entries_per_request = 1000 2021-09-17T01:56:19.064 INFO:teuthology.orchestra.run.smithi013.stdout:[osd.2] 2021-09-17T01:56:19.064 INFO:teuthology.orchestra.run.smithi013.stdout: osd_max_omap_entries_per_request = 10000 2021-09-17T01:56:19.068 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2021-09-17T01:56:19.068 INFO:teuthology.orchestra.run.smithi045.stdout: chdir = "" 2021-09-17T01:56:19.068 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-17T01:56:19.068 INFO:teuthology.orchestra.run.smithi045.stdout: auth supported = cephx 2021-09-17T01:56:19.069 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.069 INFO:teuthology.orchestra.run.smithi045.stdout: filestore xattr use omap = true 2021-09-17T01:56:19.069 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.069 INFO:teuthology.orchestra.run.smithi045.stdout: mon clock drift allowed = 1.000 2021-09-17T01:56:19.069 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.070 INFO:teuthology.orchestra.run.smithi045.stdout: osd crush chooseleaf type = 0 2021-09-17T01:56:19.070 INFO:teuthology.orchestra.run.smithi045.stdout: auth debug = true 2021-09-17T01:56:19.070 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.070 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on old message = true 2021-09-17T01:56:19.070 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on bug = true 2021-09-17T01:56:19.071 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.071 INFO:teuthology.orchestra.run.smithi045.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-17T01:56:19.071 INFO:teuthology.orchestra.run.smithi045.stdout: mon pg warn max object skew = 0 2021-09-17T01:56:19.071 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.072 INFO:teuthology.orchestra.run.smithi045.stdout: # disable pg_autoscaler by default for new pools 2021-09-17T01:56:19.072 INFO:teuthology.orchestra.run.smithi045.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-17T01:56:19.072 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.072 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default size = 2 2021-09-17T01:56:19.072 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.072 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow primary affinity = true 2021-09-17T01:56:19.073 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow pg remap = true 2021-09-17T01:56:19.073 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on legacy crush tunables = false 2021-09-17T01:56:19.073 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on crush straw calc version zero = false 2021-09-17T01:56:19.073 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on no sortbitwise = false 2021-09-17T01:56:19.074 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on osd down out interval zero = false 2021-09-17T01:56:19.074 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on too few osds = false 2021-09-17T01:56:19.074 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-17T01:56:19.074 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-17T01:56:19.074 INFO:teuthology.orchestra.run.smithi045.stdout: mon_allow_pool_size_one = true 2021-09-17T01:56:19.075 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.075 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-17T01:56:19.075 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.075 INFO:teuthology.orchestra.run.smithi045.stdout: osd default data pool replay window = 5 2021-09-17T01:56:19.075 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: mon allow pool delete = true 2021-09-17T01:56:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: mon cluster log file level = debug 2021-09-17T01:56:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: debug asserts on shutdown = true 2021-09-17T01:56:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: mon health detail to clog = false 2021-09-17T01:56:19.076 INFO:teuthology.orchestra.run.smithi045.stdout: mon host = 172.21.15.45 2021-09-17T01:56:19.077 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = e009c937-b2b5-4685-bce2-3ad003fa6b2b 2021-09-17T01:56:19.077 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.077 INFO:teuthology.orchestra.run.smithi045.stdout:[osd] 2021-09-17T01:56:19.077 INFO:teuthology.orchestra.run.smithi045.stdout: osd journal size = 100 2021-09-17T01:56:19.077 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.078 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub load threshold = 5.0 2021-09-17T01:56:19.078 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub max interval = 600 2021-09-17T01:56:19.078 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.078 INFO:teuthology.orchestra.run.smithi045.stdout: osd recover clone overlap = true 2021-09-17T01:56:19.078 INFO:teuthology.orchestra.run.smithi045.stdout: osd recovery max chunk = 1048576 2021-09-17T01:56:19.079 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.079 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug shutdown = true 2021-09-17T01:56:19.079 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug op order = true 2021-09-17T01:56:19.079 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify stray on activate = true 2021-09-17T01:56:19.079 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.080 INFO:teuthology.orchestra.run.smithi045.stdout: osd open classes on start = true 2021-09-17T01:56:19.080 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug pg log writeout = true 2021-09-17T01:56:19.080 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.080 INFO:teuthology.orchestra.run.smithi045.stdout: osd deep scrub update digest min age = 30 2021-09-17T01:56:19.080 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.081 INFO:teuthology.orchestra.run.smithi045.stdout: osd map max advance = 10 2021-09-17T01:56:19.081 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.081 INFO:teuthology.orchestra.run.smithi045.stdout: journal zero on create = true 2021-09-17T01:56:19.081 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.081 INFO:teuthology.orchestra.run.smithi045.stdout: filestore ondisk finisher threads = 3 2021-09-17T01:56:19.082 INFO:teuthology.orchestra.run.smithi045.stdout: filestore apply finisher threads = 3 2021-09-17T01:56:19.082 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.082 INFO:teuthology.orchestra.run.smithi045.stdout: bdev debug aio = true 2021-09-17T01:56:19.082 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug misdirected ops = true 2021-09-17T01:56:19.082 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2021-09-17T01:56:19.083 INFO:teuthology.orchestra.run.smithi045.stdout: debug osd = 20 2021-09-17T01:56:19.083 INFO:teuthology.orchestra.run.smithi045.stdout: rocksdb delete range threshold = 0 2021-09-17T01:56:19.083 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.083 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr] 2021-09-17T01:56:19.083 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2021-09-17T01:56:19.084 INFO:teuthology.orchestra.run.smithi045.stdout: debug mgr = 20 2021-09-17T01:56:19.084 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2021-09-17T01:56:19.084 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2021-09-17T01:56:19.084 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2021-09-17T01:56:19.084 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2021-09-17T01:56:19.084 INFO:teuthology.orchestra.run.smithi045.stdout: mgr/telemetry/nag = false 2021-09-17T01:56:19.085 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.085 INFO:teuthology.orchestra.run.smithi045.stdout:[mon] 2021-09-17T01:56:19.085 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2021-09-17T01:56:19.085 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2021-09-17T01:56:19.085 INFO:teuthology.orchestra.run.smithi045.stdout: debug paxos = 20 2021-09-17T01:56:19.086 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2021-09-17T01:56:19.086 INFO:teuthology.orchestra.run.smithi045.stdout: mon data avail warn = 5 2021-09-17T01:56:19.086 INFO:teuthology.orchestra.run.smithi045.stdout: mon mgr mkfs grace = 240 2021-09-17T01:56:19.086 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2021-09-17T01:56:19.086 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd reporter subtree level = osd 2021-09-17T01:56:19.087 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd prime pg temp = true 2021-09-17T01:56:19.087 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2021-09-17T01:56:19.087 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.087 INFO:teuthology.orchestra.run.smithi045.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-17T01:56:19.087 INFO:teuthology.orchestra.run.smithi045.stdout: auth mon ticket ttl = 660 # 11m 2021-09-17T01:56:19.088 INFO:teuthology.orchestra.run.smithi045.stdout: auth service ticket ttl = 240 # 4m 2021-09-17T01:56:19.088 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.088 INFO:teuthology.orchestra.run.smithi045.stdout: # don't complain about insecure global_id in the test suite 2021-09-17T01:56:19.088 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-17T01:56:19.088 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-17T01:56:19.088 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:19.089 INFO:teuthology.orchestra.run.smithi045.stdout:[client] 2021-09-17T01:56:19.089 INFO:teuthology.orchestra.run.smithi045.stdout: rgw cache enabled = true 2021-09-17T01:56:19.089 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable ops log = true 2021-09-17T01:56:19.089 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable usage log = true 2021-09-17T01:56:19.089 INFO:teuthology.orchestra.run.smithi045.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-17T01:56:19.090 INFO:teuthology.orchestra.run.smithi045.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-17T01:56:19.090 INFO:teuthology.orchestra.run.smithi045.stdout: debug rgw = 20 2021-09-17T01:56:19.090 INFO:teuthology.orchestra.run.smithi045.stdout: rgw crypt require ssl = False 2021-09-17T01:56:19.090 INFO:teuthology.orchestra.run.smithi045.stdout: rgw crypt s3 kms backend = testing 2021-09-17T01:56:19.090 INFO:teuthology.orchestra.run.smithi045.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T01:56:19.091 INFO:teuthology.orchestra.run.smithi045.stdout: rgw curl low speed time = 300 2021-09-17T01:56:19.091 INFO:teuthology.orchestra.run.smithi045.stdout: rgw data log num shards = 4 2021-09-17T01:56:19.091 INFO:teuthology.orchestra.run.smithi045.stdout: rgw md log max shards = 4 2021-09-17T01:56:19.091 INFO:teuthology.orchestra.run.smithi045.stdout: rgw sync log trim interval = 0 2021-09-17T01:56:19.091 INFO:teuthology.orchestra.run.smithi045.stdout: rgw sync meta inject err probability = 0.1 2021-09-17T01:56:19.092 INFO:teuthology.orchestra.run.smithi045.stdout: rgw sync obj etag verify = True 2021-09-17T01:56:19.092 INFO:teuthology.orchestra.run.smithi045.stdout: setgroup = ceph 2021-09-17T01:56:19.092 INFO:teuthology.orchestra.run.smithi045.stdout: setuser = ceph 2021-09-17T01:56:19.092 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.a] 2021-09-17T01:56:19.092 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.0] 2021-09-17T01:56:19.093 INFO:teuthology.orchestra.run.smithi045.stdout: osd_max_omap_entries_per_request = 10 2021-09-17T01:56:19.093 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.1] 2021-09-17T01:56:19.093 INFO:teuthology.orchestra.run.smithi045.stdout: osd_max_omap_entries_per_request = 1000 2021-09-17T01:56:19.093 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.2] 2021-09-17T01:56:19.093 INFO:teuthology.orchestra.run.smithi045.stdout: osd_max_omap_entries_per_request = 10000 2021-09-17T01:56:19.094 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-09-17T01:56:19.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-09-17T01:56:19.185 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-17T01:56:19.185 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:19.186 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2021-09-17T01:56:19.200 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:19.201 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2021-09-17T01:56:19.254 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi013.front.sepia.ceph.com 2021-09-17T01:56:19.255 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:56:19.255 DEBUG:teuthology.orchestra.run.smithi013:> sudo dd of=/etc/ceph/c2.keyring 2021-09-17T01:56:19.256 DEBUG:teuthology.orchestra.run.smithi013:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-17T01:56:19.297 DEBUG:teuthology.orchestra.run.smithi013:> set -ex 2021-09-17T01:56:19.298 DEBUG:teuthology.orchestra.run.smithi013:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-17T01:56:19.354 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi045.front.sepia.ceph.com 2021-09-17T01:56:19.355 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:19.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/c2.keyring 2021-09-17T01:56:19.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-17T01:56:19.390 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:19.391 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-17T01:56:19.445 INFO:tasks.ceph:Setting up mon nodes... 2021-09-17T01:56:19.446 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-17T01:56:19.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-09-17T01:56:19.534 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-09-17T01:56:19.536 INFO:tasks.ceph:Setting up mds nodes... 2021-09-17T01:56:19.536 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-17T01:56:19.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-09-17T01:56:19.575 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/c2.client.0.keyring 2021-09-17T01:56:19.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-09-17T01:56:19.665 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/c2.client.1.keyring 2021-09-17T01:56:19.678 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-17T01:56:19.678 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi013.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-17T01:56:19.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-09-17T01:56:19.743 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T01:56:19.743 INFO:tasks.ceph:role: c2.osd.0 2021-09-17T01:56:19.743 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com 2021-09-17T01:56:19.744 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-17T01:56:19.807 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T01:56:19.857 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T01:56:19.858 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T01:56:19.858 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 2021-09-17T01:56:19.858 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T01:56:19.859 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2021-09-17T01:56:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T01:56:19.859 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T01:56:19.859 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T01:56:19.860 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T01:56:19.861 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2021-09-17T01:56:19.861 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2021-09-17T01:56:19.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-09-17T01:56:19.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-09-17T01:56:20.009 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-09-17T01:56:20.074 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T01:56:20.075 INFO:tasks.ceph:role: c2.osd.1 2021-09-17T01:56:20.075 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com 2021-09-17T01:56:20.076 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-17T01:56:20.139 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T01:56:20.139 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T01:56:20.139 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T01:56:20.140 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 2021-09-17T01:56:20.140 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T01:56:20.140 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2021-09-17T01:56:20.140 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T01:56:20.140 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T01:56:20.141 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T01:56:20.141 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T01:56:20.161 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2021-09-17T01:56:20.163 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2021-09-17T01:56:20.163 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-09-17T01:56:20.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-09-17T01:56:20.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-09-17T01:56:20.373 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T01:56:20.373 INFO:tasks.ceph:role: c2.osd.2 2021-09-17T01:56:20.374 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com 2021-09-17T01:56:20.374 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-17T01:56:20.437 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T01:56:20.438 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T01:56:20.438 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T01:56:20.438 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 2021-09-17T01:56:20.438 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T01:56:20.439 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2021-09-17T01:56:20.439 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T01:56:20.439 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T01:56:20.439 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T01:56:20.439 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T01:56:20.460 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2021-09-17T01:56:20.461 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2021-09-17T01:56:20.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-09-17T01:56:20.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-09-17T01:56:20.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-17T01:56:20.689 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:20.687+0000 7f937fd4e0c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-09-17T01:56:20.690 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:20.687+0000 7f937fd4e0c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-09-17T01:56:20.690 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:20.687+0000 7f937fd4e0c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-17T01:56:20.690 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:20.687+0000 7f937fd4e0c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-17T01:56:20.691 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:20.687+0000 7f937fd4e0c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-17T01:56:20.691 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:20.687+0000 7f937fd4e0c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-09-17T01:56:22.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-09-17T01:56:22.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-17T01:56:22.930 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:22.928+0000 7fb95816a0c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-09-17T01:56:22.931 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:22.928+0000 7fb95816a0c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-09-17T01:56:22.931 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:22.928+0000 7fb95816a0c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-17T01:56:22.931 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:22.928+0000 7fb95816a0c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-17T01:56:22.932 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:22.928+0000 7fb95816a0c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-17T01:56:22.932 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:22.928+0000 7fb95816a0c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-09-17T01:56:25.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-09-17T01:56:25.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-17T01:56:25.135 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:25.134+0000 7f384b2120c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-09-17T01:56:25.135 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:25.134+0000 7f384b2120c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-09-17T01:56:25.135 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:25.134+0000 7f384b2120c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-17T01:56:25.136 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:25.134+0000 7f384b2120c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-17T01:56:25.136 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:25.134+0000 7f384b2120c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-17T01:56:25.136 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:56:25.134+0000 7f384b2120c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-09-17T01:56:27.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-09-17T01:56:27.249 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-17T01:56:27.249 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:27.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2021-09-17T01:56:27.317 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:27.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2021-09-17T01:56:27.385 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:27.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2021-09-17T01:56:27.453 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:27.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2021-09-17T01:56:27.519 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:27.520 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2021-09-17T01:56:27.576 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:27.576 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2021-09-17T01:56:27.631 INFO:tasks.ceph:Adding keys to all mons... 2021-09-17T01:56:27.631 DEBUG:teuthology.orchestra.run.smithi045:> sudo tee -a /etc/ceph/c2.keyring 2021-09-17T01:56:27.695 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.x] 2021-09-17T01:56:27.696 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDD9UNhP+7PHxAAae+b3kVafpiELjpCaTQwcg== 2021-09-17T01:56:27.696 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.0] 2021-09-17T01:56:27.696 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDE9UNh3WoOKRAAy4zcTRu9AWJh0e1RufU2eA== 2021-09-17T01:56:27.697 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.1] 2021-09-17T01:56:27.697 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDG9UNhCZVtNxAANQDvNaDGSeXLciHyRO4m+w== 2021-09-17T01:56:27.697 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.2] 2021-09-17T01:56:27.697 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDJ9UNhnpMFCBAApoiRX+RxcJ1ithyP8nwDrQ== 2021-09-17T01:56:27.697 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2021-09-17T01:56:27.698 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDD9UNh35xCIhAAFDf3qy8I2fPfavCF0nuRBQ== 2021-09-17T01:56:27.698 INFO:teuthology.orchestra.run.smithi045.stdout:[client.1] 2021-09-17T01:56:27.698 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDD9UNhLWyuJxAAJZdZO4l3pwxnpPFfzsIJHw== 2021-09-17T01:56:27.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-17T01:56:27.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T01:56:27.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T01:56:27.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T01:56:27.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-17T01:56:27.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-17T01:56:28.030 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-17T01:56:28.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-09-17T01:56:28.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-09-17T01:56:28.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-09-17T01:56:28.172 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-09-17T01:56:28.227 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-09-17T01:56:28.228 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-09-17T01:56:28.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-09-17T01:56:28.270 INFO:tasks.ceph.c2.mon.a:Started 2021-09-17T01:56:28.270 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-09-17T01:56:28.271 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-09-17T01:56:28.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-09-17T01:56:28.273 INFO:tasks.ceph.c2.mgr.x:Started 2021-09-17T01:56:28.274 INFO:tasks.ceph:Setting crush tunables to default 2021-09-17T01:56:28.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster c2 osd crush tunables default 2021-09-17T01:56:28.481 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2021-09-17T01:56:28.491 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-09-17T01:56:28.492 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:28.492 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2021-09-17T01:56:28.517 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:28.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2021-09-17T01:56:28.583 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2021-09-17T01:56:28.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2021-09-17T01:56:28.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster c2 osd new 0eaa4d0e-12fb-4d63-93d0-6537c9ca0131 0 2021-09-17T01:56:28.898 INFO:teuthology.orchestra.run.smithi045.stdout:0 2021-09-17T01:56:28.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster c2 osd new b1be6fb3-68b0-4977-8b87-f45f560f92b9 1 2021-09-17T01:56:29.117 INFO:teuthology.orchestra.run.smithi045.stdout:1 2021-09-17T01:56:29.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster c2 osd new 1658b0e3-d457-446f-a520-079df76b70de 2 2021-09-17T01:56:29.339 INFO:teuthology.orchestra.run.smithi045.stdout:2 2021-09-17T01:56:29.352 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-09-17T01:56:29.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-09-17T01:56:29.355 INFO:tasks.ceph.c2.osd.0:Started 2021-09-17T01:56:29.355 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-09-17T01:56:29.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-09-17T01:56:29.358 INFO:tasks.ceph.c2.osd.1:Started 2021-09-17T01:56:29.358 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-09-17T01:56:29.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-09-17T01:56:29.361 INFO:tasks.ceph.c2.osd.2:Started 2021-09-17T01:56:29.361 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-17T01:56:29.664 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-17T01:56:29.847 INFO:teuthology.misc.health.smithi045.stdout: 2021-09-17T01:56:29.847 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":5,"fsid":"e009c937-b2b5-4685-bce2-3ad003fa6b2b","created":"2021-09-17T01:56:28.367738+0000","modified":"2021-09-17T01:56:29.337961+0000","last_up_change":"0.000000","last_in_change":"2021-09-17T01:56:29.337961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"0eaa4d0e-12fb-4d63-93d0-6537c9ca0131","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b1be6fb3-68b0-4977-8b87-f45f560f92b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1658b0e3-d457-446f-a520-079df76b70de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T01:56:29.858 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-17T01:56:29.946 INFO:tasks.ceph.c2.osd.0.smithi045.stderr:2021-09-17T01:56:29.945+0000 7f1ac85830c0 -1 Falling back to public interface 2021-09-17T01:56:29.955 INFO:tasks.ceph.c2.osd.1.smithi045.stderr:2021-09-17T01:56:29.954+0000 7f8bfa0e90c0 -1 Falling back to public interface 2021-09-17T01:56:29.957 INFO:tasks.ceph.c2.osd.2.smithi045.stderr:2021-09-17T01:56:29.956+0000 7f1c1eccb0c0 -1 Falling back to public interface 2021-09-17T01:56:32.673 INFO:tasks.ceph.c2.osd.1.smithi045.stderr:2021-09-17T01:56:32.672+0000 7f8bfa0e90c0 -1 osd.1 0 log_to_monitors {default=true} 2021-09-17T01:56:32.674 INFO:tasks.ceph.c2.osd.0.smithi045.stderr:2021-09-17T01:56:32.672+0000 7f1ac85830c0 -1 osd.0 0 log_to_monitors {default=true} 2021-09-17T01:56:32.709 INFO:tasks.ceph.c2.osd.2.smithi045.stderr:2021-09-17T01:56:32.708+0000 7f1c1eccb0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-09-17T01:56:34.421 INFO:tasks.ceph.c2.osd.1.smithi045.stderr:2021-09-17T01:56:34.420+0000 7f8bf0aa7700 -1 osd.1 0 waiting for initial osdmap 2021-09-17T01:56:34.422 INFO:tasks.ceph.c2.osd.1.smithi045.stderr:2021-09-17T01:56:34.422+0000 7f8be943a700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T01:56:34.426 INFO:tasks.ceph.c2.osd.2.smithi045.stderr:2021-09-17T01:56:34.425+0000 7f1c15689700 -1 osd.2 0 waiting for initial osdmap 2021-09-17T01:56:34.427 INFO:tasks.ceph.c2.osd.2.smithi045.stderr:2021-09-17T01:56:34.427+0000 7f1c0e01c700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T01:56:34.464 INFO:tasks.ceph.c2.osd.0.smithi045.stderr:2021-09-17T01:56:34.463+0000 7f1abef41700 -1 osd.0 0 waiting for initial osdmap 2021-09-17T01:56:34.466 INFO:tasks.ceph.c2.osd.0.smithi045.stderr:2021-09-17T01:56:34.465+0000 7f1ab78d4700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T01:56:36.161 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-17T01:56:36.468 INFO:teuthology.misc.health.smithi045.stdout: 2021-09-17T01:56:36.469 INFO:teuthology.misc.health.smithi045.stdout:{"epoch":9,"fsid":"e009c937-b2b5-4685-bce2-3ad003fa6b2b","created":"2021-09-17T01:56:28.367738+0000","modified":"2021-09-17T01:56:36.454362+0000","last_up_change":"2021-09-17T01:56:35.423772+0000","last_in_change":"2021-09-17T01:56:29.337961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-17T01:56:35.493630+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"0eaa4d0e-12fb-4d63-93d0-6537c9ca0131","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":31544},{"type":"v1","addr":"172.21.15.45:6801","nonce":31544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":31544},{"type":"v1","addr":"172.21.15.45:6803","nonce":31544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":31544},{"type":"v1","addr":"172.21.15.45:6807","nonce":31544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":31544},{"type":"v1","addr":"172.21.15.45:6805","nonce":31544}]},"public_addr":"172.21.15.45:6801/31544","cluster_addr":"172.21.15.45:6803/31544","heartbeat_back_addr":"172.21.15.45:6807/31544","heartbeat_front_addr":"172.21.15.45:6805/31544","state":["exists","up"]},{"osd":1,"uuid":"b1be6fb3-68b0-4977-8b87-f45f560f92b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":31545},{"type":"v1","addr":"172.21.15.45:6809","nonce":31545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":31545},{"type":"v1","addr":"172.21.15.45:6811","nonce":31545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":31545},{"type":"v1","addr":"172.21.15.45:6815","nonce":31545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":31545},{"type":"v1","addr":"172.21.15.45:6813","nonce":31545}]},"public_addr":"172.21.15.45:6809/31545","cluster_addr":"172.21.15.45:6811/31545","heartbeat_back_addr":"172.21.15.45:6815/31545","heartbeat_front_addr":"172.21.15.45:6813/31545","state":["exists","up"]},{"osd":2,"uuid":"1658b0e3-d457-446f-a520-079df76b70de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":31546},{"type":"v1","addr":"172.21.15.45:6817","nonce":31546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":31546},{"type":"v1","addr":"172.21.15.45:6819","nonce":31546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":31546},{"type":"v1","addr":"172.21.15.45:6823","nonce":31546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":31546},{"type":"v1","addr":"172.21.15.45:6821","nonce":31546}]},"public_addr":"172.21.15.45:6817/31546","cluster_addr":"172.21.15.45:6819/31546","heartbeat_back_addr":"172.21.15.45:6823/31546","heartbeat_front_addr":"172.21.15.45:6821/31546","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.647456+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.684427+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.667738+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T01:56:36.481 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-17T01:56:36.481 INFO:tasks.ceph:Creating RBD pool 2021-09-17T01:56:36.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster c2 osd pool create rbd 8 2021-09-17T01:56:37.464 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'rbd' created 2021-09-17T01:56:37.479 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-17T01:56:38.467 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-17T01:56:38.480 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-09-17T01:56:38.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-17T01:56:38.481 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-17T01:56:38.781 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:38.782 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":11,"fsid":"e009c937-b2b5-4685-bce2-3ad003fa6b2b","created":"2021-09-17T01:56:28.367738+0000","modified":"2021-09-17T01:56:38.455886+0000","last_up_change":"2021-09-17T01:56:35.423772+0000","last_in_change":"2021-09-17T01:56:29.337961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-17T01:56:35.493630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-17T01:56:36.809049+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0eaa4d0e-12fb-4d63-93d0-6537c9ca0131","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":31544},{"type":"v1","addr":"172.21.15.45:6801","nonce":31544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":31544},{"type":"v1","addr":"172.21.15.45:6803","nonce":31544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":31544},{"type":"v1","addr":"172.21.15.45:6807","nonce":31544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":31544},{"type":"v1","addr":"172.21.15.45:6805","nonce":31544}]},"public_addr":"172.21.15.45:6801/31544","cluster_addr":"172.21.15.45:6803/31544","heartbeat_back_addr":"172.21.15.45:6807/31544","heartbeat_front_addr":"172.21.15.45:6805/31544","state":["exists","up"]},{"osd":1,"uuid":"b1be6fb3-68b0-4977-8b87-f45f560f92b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":31545},{"type":"v1","addr":"172.21.15.45:6809","nonce":31545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":31545},{"type":"v1","addr":"172.21.15.45:6811","nonce":31545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":31545},{"type":"v1","addr":"172.21.15.45:6815","nonce":31545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":31545},{"type":"v1","addr":"172.21.15.45:6813","nonce":31545}]},"public_addr":"172.21.15.45:6809/31545","cluster_addr":"172.21.15.45:6811/31545","heartbeat_back_addr":"172.21.15.45:6815/31545","heartbeat_front_addr":"172.21.15.45:6813/31545","state":["exists","up"]},{"osd":2,"uuid":"1658b0e3-d457-446f-a520-079df76b70de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":31546},{"type":"v1","addr":"172.21.15.45:6817","nonce":31546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":31546},{"type":"v1","addr":"172.21.15.45:6819","nonce":31546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":31546},{"type":"v1","addr":"172.21.15.45:6823","nonce":31546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":31546},{"type":"v1","addr":"172.21.15.45:6821","nonce":31546}]},"public_addr":"172.21.15.45:6817/31546","cluster_addr":"172.21.15.45:6819/31546","heartbeat_back_addr":"172.21.15.45:6823/31546","heartbeat_front_addr":"172.21.15.45:6821/31546","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.647456+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.684427+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.667738+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T01:56:38.795 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2021-09-17T01:56:35.493630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-09-17T01:56:36.809049+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-09-17T01:56:38.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get .mgr pg_num 2021-09-17T01:56:39.099 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2021-09-17T01:56:39.113 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2021-09-17T01:56:39.416 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 8 2021-09-17T01:56:39.429 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-09-17T01:56:39.430 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-09-17T01:56:39.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2021-09-17T01:56:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:39.791 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":31225},{"type":"v1","addr":"172.21.15.45:6825","nonce":31225}]},"active_addr":"172.21.15.45:6825/31225","active_change":"2021-09-17T01:56:33.446373+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":281888535}]},{"addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":2499717109}]},{"addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":3393386605}]}]}} 2021-09-17T01:56:39.793 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-09-17T01:56:39.793 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-09-17T01:56:39.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-17T01:56:40.094 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:40.094 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":12,"fsid":"e009c937-b2b5-4685-bce2-3ad003fa6b2b","created":"2021-09-17T01:56:28.367738+0000","modified":"2021-09-17T01:56:39.458586+0000","last_up_change":"2021-09-17T01:56:35.423772+0000","last_in_change":"2021-09-17T01:56:29.337961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-17T01:56:35.493630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-17T01:56:36.809049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0eaa4d0e-12fb-4d63-93d0-6537c9ca0131","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":31544},{"type":"v1","addr":"172.21.15.45:6801","nonce":31544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":31544},{"type":"v1","addr":"172.21.15.45:6803","nonce":31544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":31544},{"type":"v1","addr":"172.21.15.45:6807","nonce":31544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":31544},{"type":"v1","addr":"172.21.15.45:6805","nonce":31544}]},"public_addr":"172.21.15.45:6801/31544","cluster_addr":"172.21.15.45:6803/31544","heartbeat_back_addr":"172.21.15.45:6807/31544","heartbeat_front_addr":"172.21.15.45:6805/31544","state":["exists","up"]},{"osd":1,"uuid":"b1be6fb3-68b0-4977-8b87-f45f560f92b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":31545},{"type":"v1","addr":"172.21.15.45:6809","nonce":31545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":31545},{"type":"v1","addr":"172.21.15.45:6811","nonce":31545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":31545},{"type":"v1","addr":"172.21.15.45:6815","nonce":31545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":31545},{"type":"v1","addr":"172.21.15.45:6813","nonce":31545}]},"public_addr":"172.21.15.45:6809/31545","cluster_addr":"172.21.15.45:6811/31545","heartbeat_back_addr":"172.21.15.45:6815/31545","heartbeat_front_addr":"172.21.15.45:6813/31545","state":["exists","up"]},{"osd":2,"uuid":"1658b0e3-d457-446f-a520-079df76b70de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":31546},{"type":"v1","addr":"172.21.15.45:6817","nonce":31546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":31546},{"type":"v1","addr":"172.21.15.45:6819","nonce":31546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":31546},{"type":"v1","addr":"172.21.15.45:6823","nonce":31546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":31546},{"type":"v1","addr":"172.21.15.45:6821","nonce":31546}]},"public_addr":"172.21.15.45:6817/31546","cluster_addr":"172.21.15.45:6819/31546","heartbeat_back_addr":"172.21.15.45:6823/31546","heartbeat_front_addr":"172.21.15.45:6821/31546","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.647456+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.684427+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.667738+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T01:56:40.107 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-09-17T01:56:40.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-17T01:56:40.409 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:40.410 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":12,"fsid":"e009c937-b2b5-4685-bce2-3ad003fa6b2b","created":"2021-09-17T01:56:28.367738+0000","modified":"2021-09-17T01:56:39.458586+0000","last_up_change":"2021-09-17T01:56:35.423772+0000","last_in_change":"2021-09-17T01:56:29.337961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-17T01:56:35.493630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-17T01:56:36.809049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0eaa4d0e-12fb-4d63-93d0-6537c9ca0131","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":31544},{"type":"v1","addr":"172.21.15.45:6801","nonce":31544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":31544},{"type":"v1","addr":"172.21.15.45:6803","nonce":31544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":31544},{"type":"v1","addr":"172.21.15.45:6807","nonce":31544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":31544},{"type":"v1","addr":"172.21.15.45:6805","nonce":31544}]},"public_addr":"172.21.15.45:6801/31544","cluster_addr":"172.21.15.45:6803/31544","heartbeat_back_addr":"172.21.15.45:6807/31544","heartbeat_front_addr":"172.21.15.45:6805/31544","state":["exists","up"]},{"osd":1,"uuid":"b1be6fb3-68b0-4977-8b87-f45f560f92b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":31545},{"type":"v1","addr":"172.21.15.45:6809","nonce":31545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":31545},{"type":"v1","addr":"172.21.15.45:6811","nonce":31545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":31545},{"type":"v1","addr":"172.21.15.45:6815","nonce":31545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":31545},{"type":"v1","addr":"172.21.15.45:6813","nonce":31545}]},"public_addr":"172.21.15.45:6809/31545","cluster_addr":"172.21.15.45:6811/31545","heartbeat_back_addr":"172.21.15.45:6815/31545","heartbeat_front_addr":"172.21.15.45:6813/31545","state":["exists","up"]},{"osd":2,"uuid":"1658b0e3-d457-446f-a520-079df76b70de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":31546},{"type":"v1","addr":"172.21.15.45:6817","nonce":31546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":31546},{"type":"v1","addr":"172.21.15.45:6819","nonce":31546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":31546},{"type":"v1","addr":"172.21.15.45:6823","nonce":31546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":31546},{"type":"v1","addr":"172.21.15.45:6821","nonce":31546}]},"public_addr":"172.21.15.45:6817/31546","cluster_addr":"172.21.15.45:6819/31546","heartbeat_back_addr":"172.21.15.45:6823/31546","heartbeat_front_addr":"172.21.15.45:6821/31546","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.647456+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.684427+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T01:56:33.667738+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T01:56:40.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2021-09-17T01:56:40.573 INFO:teuthology.orchestra.run.smithi045.stdout:34359738372 2021-09-17T01:56:40.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2021-09-17T01:56:40.724 INFO:teuthology.orchestra.run.smithi045.stdout:34359738372 2021-09-17T01:56:40.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2021-09-17T01:56:40.874 INFO:teuthology.orchestra.run.smithi045.stdout:34359738371 2021-09-17T01:56:40.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-09-17T01:56:41.174 INFO:teuthology.orchestra.run.smithi045.stdout:34359738370 2021-09-17T01:56:41.187 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738370 for osd.0 2021-09-17T01:56:42.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-09-17T01:56:42.508 INFO:teuthology.orchestra.run.smithi045.stdout:34359738372 2021-09-17T01:56:42.521 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.0 2021-09-17T01:56:42.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-09-17T01:56:42.824 INFO:teuthology.orchestra.run.smithi045.stdout:34359738372 2021-09-17T01:56:42.837 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2021-09-17T01:56:42.837 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-09-17T01:56:43.135 INFO:teuthology.orchestra.run.smithi045.stdout:34359738371 2021-09-17T01:56:43.148 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2021-09-17T01:56:43.148 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-09-17T01:56:43.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-17T01:56:43.447 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:43.449 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2021-09-17T01:56:43.460 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-09-17T01:56:41.451391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24964,"kb_used_data":1460,"kb_used_omap":0,"kb_used_meta":23104,"kb_avail":314547836,"statfs":{"total":322122547200,"available":322096984064,"internally_reserved":0,"allocated":1495040,"data_stored":1263223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23658496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.464678+0000","last_change":"2021-09-17T01:56:38.464063+0000","last_active":"2021-09-17T01:56:38.464678+0000","last_peered":"2021-09-17T01:56:38.464678+0000","last_clean":"2021-09-17T01:56:38.464678+0000","last_became_active":"2021-09-17T01:56:38.463758+0000","last_became_peered":"2021-09-17T01:56:38.463758+0000","last_unstale":"2021-09-17T01:56:38.464678+0000","last_undegraded":"2021-09-17T01:56:38.464678+0000","last_fullsized":"2021-09-17T01:56:38.464678+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.464717+0000","last_change":"2021-09-17T01:56:38.464131+0000","last_active":"2021-09-17T01:56:38.464717+0000","last_peered":"2021-09-17T01:56:38.464717+0000","last_clean":"2021-09-17T01:56:38.464717+0000","last_became_active":"2021-09-17T01:56:38.463857+0000","last_became_peered":"2021-09-17T01:56:38.463857+0000","last_unstale":"2021-09-17T01:56:38.464717+0000","last_undegraded":"2021-09-17T01:56:38.464717+0000","last_fullsized":"2021-09-17T01:56:38.464717+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.466644+0000","last_change":"2021-09-17T01:56:38.465814+0000","last_active":"2021-09-17T01:56:38.466644+0000","last_peered":"2021-09-17T01:56:38.466644+0000","last_clean":"2021-09-17T01:56:38.466644+0000","last_became_active":"2021-09-17T01:56:38.465577+0000","last_became_peered":"2021-09-17T01:56:38.465577+0000","last_unstale":"2021-09-17T01:56:38.466644+0000","last_undegraded":"2021-09-17T01:56:38.466644+0000","last_fullsized":"2021-09-17T01:56:38.466644+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.466858+0000","last_change":"2021-09-17T01:56:38.466241+0000","last_active":"2021-09-17T01:56:38.466858+0000","last_peered":"2021-09-17T01:56:38.466858+0000","last_clean":"2021-09-17T01:56:38.466858+0000","last_became_active":"2021-09-17T01:56:38.465894+0000","last_became_peered":"2021-09-17T01:56:38.465894+0000","last_unstale":"2021-09-17T01:56:38.466858+0000","last_undegraded":"2021-09-17T01:56:38.466858+0000","last_fullsized":"2021-09-17T01:56:38.466858+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.460170+0000","last_change":"2021-09-17T01:56:37.464126+0000","last_active":"2021-09-17T01:56:38.460170+0000","last_peered":"2021-09-17T01:56:38.460170+0000","last_clean":"2021-09-17T01:56:38.460170+0000","last_became_active":"2021-09-17T01:56:37.463880+0000","last_became_peered":"2021-09-17T01:56:37.463880+0000","last_unstale":"2021-09-17T01:56:38.460170+0000","last_undegraded":"2021-09-17T01:56:38.460170+0000","last_fullsized":"2021-09-17T01:56:38.460170+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:36.454362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:36.454362+0000","last_clean_scrub_stamp":"2021-09-17T01:56:36.454362+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.464707+0000","last_change":"2021-09-17T01:56:38.463047+0000","last_active":"2021-09-17T01:56:38.464707+0000","last_peered":"2021-09-17T01:56:38.464707+0000","last_clean":"2021-09-17T01:56:38.464707+0000","last_became_active":"2021-09-17T01:56:38.462739+0000","last_became_peered":"2021-09-17T01:56:38.462739+0000","last_unstale":"2021-09-17T01:56:38.464707+0000","last_undegraded":"2021-09-17T01:56:38.464707+0000","last_fullsized":"2021-09-17T01:56:38.464707+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.465758+0000","last_change":"2021-09-17T01:56:38.465238+0000","last_active":"2021-09-17T01:56:38.465758+0000","last_peered":"2021-09-17T01:56:38.465758+0000","last_clean":"2021-09-17T01:56:38.465758+0000","last_became_active":"2021-09-17T01:56:38.465023+0000","last_became_peered":"2021-09-17T01:56:38.465023+0000","last_unstale":"2021-09-17T01:56:38.465758+0000","last_undegraded":"2021-09-17T01:56:38.465758+0000","last_fullsized":"2021-09-17T01:56:38.465758+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.465317+0000","last_change":"2021-09-17T01:56:38.465317+0000","last_active":"2021-09-17T01:56:38.465317+0000","last_peered":"2021-09-17T01:56:38.465317+0000","last_clean":"2021-09-17T01:56:38.465317+0000","last_became_active":"2021-09-17T01:56:38.465016+0000","last_became_peered":"2021-09-17T01:56:38.465016+0000","last_unstale":"2021-09-17T01:56:38.465317+0000","last_undegraded":"2021-09-17T01:56:38.465317+0000","last_fullsized":"2021-09-17T01:56:38.465317+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.465257+0000","last_change":"2021-09-17T01:56:38.464850+0000","last_active":"2021-09-17T01:56:38.465257+0000","last_peered":"2021-09-17T01:56:38.465257+0000","last_clean":"2021-09-17T01:56:38.465257+0000","last_became_active":"2021-09-17T01:56:38.464091+0000","last_became_peered":"2021-09-17T01:56:38.464091+0000","last_unstale":"2021-09-17T01:56:38.465257+0000","last_undegraded":"2021-09-17T01:56:38.465257+0000","last_fullsized":"2021-09-17T01:56:38.465257+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7532,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":7424,"kb_avail":104850068,"statfs":{"total":107374182400,"available":107366469632,"internally_reserved":0,"allocated":102400,"data_stored":28019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7602176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8680,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104848920,"statfs":{"total":107374182400,"available":107365294080,"internally_reserved":0,"allocated":692224,"data_stored":615877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8752,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848848,"statfs":{"total":107374182400,"available":107365220352,"internally_reserved":0,"allocated":700416,"data_stored":619327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-17T01:56:43.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-17T01:56:43.759 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:43.761 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2021-09-17T01:56:43.772 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2021-09-17T01:56:43.451689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24964,"kb_used_data":1460,"kb_used_omap":0,"kb_used_meta":23104,"kb_avail":314547836,"statfs":{"total":322122547200,"available":322096984064,"internally_reserved":0,"allocated":1495040,"data_stored":1263223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23658496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000298"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.464678+0000","last_change":"2021-09-17T01:56:38.464063+0000","last_active":"2021-09-17T01:56:38.464678+0000","last_peered":"2021-09-17T01:56:38.464678+0000","last_clean":"2021-09-17T01:56:38.464678+0000","last_became_active":"2021-09-17T01:56:38.463758+0000","last_became_peered":"2021-09-17T01:56:38.463758+0000","last_unstale":"2021-09-17T01:56:38.464678+0000","last_undegraded":"2021-09-17T01:56:38.464678+0000","last_fullsized":"2021-09-17T01:56:38.464678+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.464717+0000","last_change":"2021-09-17T01:56:38.464131+0000","last_active":"2021-09-17T01:56:38.464717+0000","last_peered":"2021-09-17T01:56:38.464717+0000","last_clean":"2021-09-17T01:56:38.464717+0000","last_became_active":"2021-09-17T01:56:38.463857+0000","last_became_peered":"2021-09-17T01:56:38.463857+0000","last_unstale":"2021-09-17T01:56:38.464717+0000","last_undegraded":"2021-09-17T01:56:38.464717+0000","last_fullsized":"2021-09-17T01:56:38.464717+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.466644+0000","last_change":"2021-09-17T01:56:38.465814+0000","last_active":"2021-09-17T01:56:38.466644+0000","last_peered":"2021-09-17T01:56:38.466644+0000","last_clean":"2021-09-17T01:56:38.466644+0000","last_became_active":"2021-09-17T01:56:38.465577+0000","last_became_peered":"2021-09-17T01:56:38.465577+0000","last_unstale":"2021-09-17T01:56:38.466644+0000","last_undegraded":"2021-09-17T01:56:38.466644+0000","last_fullsized":"2021-09-17T01:56:38.466644+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.466858+0000","last_change":"2021-09-17T01:56:38.466241+0000","last_active":"2021-09-17T01:56:38.466858+0000","last_peered":"2021-09-17T01:56:38.466858+0000","last_clean":"2021-09-17T01:56:38.466858+0000","last_became_active":"2021-09-17T01:56:38.465894+0000","last_became_peered":"2021-09-17T01:56:38.465894+0000","last_unstale":"2021-09-17T01:56:38.466858+0000","last_undegraded":"2021-09-17T01:56:38.466858+0000","last_fullsized":"2021-09-17T01:56:38.466858+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.460170+0000","last_change":"2021-09-17T01:56:37.464126+0000","last_active":"2021-09-17T01:56:38.460170+0000","last_peered":"2021-09-17T01:56:38.460170+0000","last_clean":"2021-09-17T01:56:38.460170+0000","last_became_active":"2021-09-17T01:56:37.463880+0000","last_became_peered":"2021-09-17T01:56:37.463880+0000","last_unstale":"2021-09-17T01:56:38.460170+0000","last_undegraded":"2021-09-17T01:56:38.460170+0000","last_fullsized":"2021-09-17T01:56:38.460170+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:36.454362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:36.454362+0000","last_clean_scrub_stamp":"2021-09-17T01:56:36.454362+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.464707+0000","last_change":"2021-09-17T01:56:38.463047+0000","last_active":"2021-09-17T01:56:38.464707+0000","last_peered":"2021-09-17T01:56:38.464707+0000","last_clean":"2021-09-17T01:56:38.464707+0000","last_became_active":"2021-09-17T01:56:38.462739+0000","last_became_peered":"2021-09-17T01:56:38.462739+0000","last_unstale":"2021-09-17T01:56:38.464707+0000","last_undegraded":"2021-09-17T01:56:38.464707+0000","last_fullsized":"2021-09-17T01:56:38.464707+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.465758+0000","last_change":"2021-09-17T01:56:38.465238+0000","last_active":"2021-09-17T01:56:38.465758+0000","last_peered":"2021-09-17T01:56:38.465758+0000","last_clean":"2021-09-17T01:56:38.465758+0000","last_became_active":"2021-09-17T01:56:38.465023+0000","last_became_peered":"2021-09-17T01:56:38.465023+0000","last_unstale":"2021-09-17T01:56:38.465758+0000","last_undegraded":"2021-09-17T01:56:38.465758+0000","last_fullsized":"2021-09-17T01:56:38.465758+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.465317+0000","last_change":"2021-09-17T01:56:38.465317+0000","last_active":"2021-09-17T01:56:38.465317+0000","last_peered":"2021-09-17T01:56:38.465317+0000","last_clean":"2021-09-17T01:56:38.465317+0000","last_became_active":"2021-09-17T01:56:38.465016+0000","last_became_peered":"2021-09-17T01:56:38.465016+0000","last_unstale":"2021-09-17T01:56:38.465317+0000","last_undegraded":"2021-09-17T01:56:38.465317+0000","last_fullsized":"2021-09-17T01:56:38.465317+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-17T01:56:38.465257+0000","last_change":"2021-09-17T01:56:38.464850+0000","last_active":"2021-09-17T01:56:38.465257+0000","last_peered":"2021-09-17T01:56:38.465257+0000","last_clean":"2021-09-17T01:56:38.465257+0000","last_became_active":"2021-09-17T01:56:38.464091+0000","last_became_peered":"2021-09-17T01:56:38.464091+0000","last_unstale":"2021-09-17T01:56:38.465257+0000","last_undegraded":"2021-09-17T01:56:38.465257+0000","last_fullsized":"2021-09-17T01:56:38.465257+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T01:56:37.454859+0000","last_clean_scrub_stamp":"2021-09-17T01:56:37.454859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7532,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":7424,"kb_avail":104850068,"statfs":{"total":107374182400,"available":107366469632,"internally_reserved":0,"allocated":102400,"data_stored":28019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7602176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8680,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104848920,"statfs":{"total":107374182400,"available":107365294080,"internally_reserved":0,"allocated":692224,"data_stored":615877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8752,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848848,"statfs":{"total":107374182400,"available":107365220352,"internally_reserved":0,"allocated":700416,"data_stored":619327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-17T01:56:43.773 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-09-17T01:56:43.773 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-09-17T01:56:43.773 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-09-17T01:56:43.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2021-09-17T01:56:44.137 INFO:teuthology.orchestra.run.smithi045.stdout: 2021-09-17T01:56:44.137 INFO:teuthology.orchestra.run.smithi045.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-17T01:56:44.150 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-09-17T01:56:44.151 INFO:teuthology.run_tasks:Running task rgw... 2021-09-17T01:56:44.167 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-09-17T01:56:44.168 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-09-17T01:56:44.168 INFO:tasks.rgw:Creating data pools 2021-09-17T01:56:44.169 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-09-17T01:56:44.169 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-17T01:56:45.241 INFO:teuthology.orchestra.run.smithi013.stderr:pool 'default.rgw.buckets.data' created 2021-09-17T01:56:45.257 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-17T01:56:46.263 INFO:teuthology.orchestra.run.smithi013.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-17T01:56:46.277 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-17T01:56:47.246 INFO:teuthology.orchestra.run.smithi013.stderr:pool 'default.rgw.buckets.index' created 2021-09-17T01:56:47.262 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-17T01:56:48.251 INFO:teuthology.orchestra.run.smithi013.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-17T01:56:48.270 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-09-17T01:56:48.270 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-17T01:56:48.574 INFO:teuthology.orchestra.run.smithi013.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-17T01:56:48.585 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-17T01:56:49.244 INFO:teuthology.orchestra.run.smithi013.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-17T01:56:49.256 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-17T01:56:49.553 INFO:teuthology.orchestra.run.smithi013.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-17T01:56:49.563 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-17T01:56:50.245 INFO:teuthology.orchestra.run.smithi013.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-17T01:56:50.257 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-09-17T01:56:50.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-17T01:56:51.487 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'default.rgw.buckets.data' created 2021-09-17T01:56:51.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-17T01:56:52.510 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-17T01:56:52.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-17T01:56:53.497 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'default.rgw.buckets.index' created 2021-09-17T01:56:53.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-17T01:56:54.512 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-17T01:56:54.529 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-09-17T01:56:54.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-17T01:56:54.828 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-17T01:56:54.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-17T01:56:55.499 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-17T01:56:55.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-17T01:56:55.809 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-17T01:56:55.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-17T01:56:56.501 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-17T01:56:56.514 DEBUG:tasks.rgw:Pools created 2021-09-17T01:56:56.514 INFO:tasks.rgw:Configuring compression type = random 2021-09-17T01:56:56.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-09-17T01:56:56.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-09-17T01:56:56.516 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-09-17T01:56:56.588 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:56:56.589 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:56:56.595 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:56.594+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.242 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.241+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.243 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.241+0000 7fb45d2fe380 20 realm 2021-09-17T01:56:58.243 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.241+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.245 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.243+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.245 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.243+0000 7fb45d2fe380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-17T01:56:58.245 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.243+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.246 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.243+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.246 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.243+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.251 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.249+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.251 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.249+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.252 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.250+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.252 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.250+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.252 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.250+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.252 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.250+0000 7fb45d2fe380 10 Creating default zonegroup 2021-09-17T01:56:58.253 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.250+0000 7fb45d2fe380 20 get_system_obj_state: rctx=0x7ffd72ac8de0 obj=:.pools.avail state=0x561792bd55c0 s->prefetch_data=0 2021-09-17T01:56:58.253 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.251+0000 7fb45d2fe380 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-17T01:56:58.264 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.262+0000 7fb45d2fe380 10 WARNING: store->list_zones() returned r=-2 2021-09-17T01:56:58.264 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.262+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.264 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.263+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.266 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.265+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.267 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.265+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.267 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.265+0000 7fb45d2fe380 10 could not read realm id: (2) No such file or directory 2021-09-17T01:56:58.268 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.265+0000 7fb45d2fe380 10 WARNING: failed to set zone as default, r=-22 2021-09-17T01:56:58.268 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.265+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.268 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.265+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.269 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.267+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.270 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.268+0000 7fb45d2fe380 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T01:56:58.270 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.268+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.270 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.268+0000 7fb45d2fe380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:56:58.271 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.268+0000 7fb45d2fe380 10 Using default name default 2021-09-17T01:56:58.271 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.268+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.271 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.269+0000 7fb45d2fe380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:56:58.271 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.269+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.272 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.269+0000 7fb45d2fe380 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T01:56:58.273 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.269+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.273 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.269+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.273 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.269+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.273 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.269+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.274 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.269+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.274 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.270+0000 7fb45d2fe380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:56:58.280 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.278+0000 7fb45d2fe380 20 RGWRados::pool_iterate: got zone_info.ab0e682e-dc0c-48fc-b378-7171369d6437 2021-09-17T01:56:58.280 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.278+0000 7fb45d2fe380 20 RGWRados::pool_iterate: got zonegroup_info.da0d806e-dc17-459c-bcd4-045d23fda180 2021-09-17T01:56:58.281 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.278+0000 7fb45d2fe380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T01:56:58.281 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.278+0000 7fb45d2fe380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T01:56:58.281 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.278+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.281 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.279+0000 7fb45d2fe380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:56:58.281 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.279+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.283 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.279+0000 7fb45d2fe380 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T01:56:58.283 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.279+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.283 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.279+0000 7fb45d2fe380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:56:58.284 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.279+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.284 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:56:58.284 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.284 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.285 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 20 zone default found 2021-09-17T01:56:58.285 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 1 Realm: () 2021-09-17T01:56:58.285 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 1 ZoneGroup: default (da0d806e-dc17-459c-bcd4-045d23fda180) 2021-09-17T01:56:58.285 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 1 Zone: default (ab0e682e-dc0c-48fc-b378-7171369d6437) 2021-09-17T01:56:58.286 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-17T01:56:58.286 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 20 zonegroup default 2021-09-17T01:56:58.286 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:56:58.286 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:56:58.286 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:56:58.280+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:57:00.258 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:00.256+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:00.259 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:00.256+0000 7fb45d2fe380 20 rados->read ofs=0 len=0 2021-09-17T01:57:00.259 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:00.256+0000 7fb45d2fe380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:00.259 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:00.256+0000 7fb45d2fe380 20 started sync module instance, tier type = 2021-09-17T01:57:00.260 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:00.256+0000 7fb45d2fe380 20 started zone id=ab0e682e-dc0c-48fc-b378-7171369d6437 (name=default) with tier type = 2021-09-17T01:57:02.275 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.274+0000 7fb45d2fe380 20 add_watcher() i=0 2021-09-17T01:57:02.276 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.274+0000 7fb45d2fe380 20 add_watcher() i=1 2021-09-17T01:57:02.276 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.274+0000 7fb45d2fe380 20 add_watcher() i=2 2021-09-17T01:57:02.276 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.274+0000 7fb45d2fe380 20 add_watcher() i=3 2021-09-17T01:57:02.276 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.274+0000 7fb45d2fe380 20 add_watcher() i=4 2021-09-17T01:57:02.277 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.274+0000 7fb45d2fe380 20 add_watcher() i=5 2021-09-17T01:57:02.277 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.274+0000 7fb45d2fe380 20 add_watcher() i=6 2021-09-17T01:57:02.278 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.275+0000 7fb45d2fe380 20 add_watcher() i=7 2021-09-17T01:57:02.279 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.275+0000 7fb45d2fe380 2 all 8 watchers are set, enabling cache 2021-09-17T01:57:02.279 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.275+0000 7fb45d2fe380 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-17T01:57:02.279 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.275+0000 7fb45d2fe380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-17T01:57:02.279 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.275+0000 7fb45d2fe380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T01:57:02.280 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.275+0000 7fb45d2fe380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-17T01:57:02.280 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.275+0000 7fb45d2fe380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-17T01:57:02.281 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.275+0000 7fb45d2fe380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T01:57:02.281 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.275+0000 7fb45d2fe380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-17T01:57:02.282 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.275+0000 7fb45d2fe380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-17T01:57:02.282 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.275+0000 7fb45d2fe380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T01:57:02.283 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.276+0000 7fb45d2fe380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-17T01:57:02.283 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.276+0000 7fb45d2fe380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-17T01:57:02.283 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.276+0000 7fb45d2fe380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T01:57:02.284 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.276+0000 7fb45d2fe380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-17T01:57:02.284 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.276+0000 7fb45d2fe380 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-17T01:57:02.286 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.277+0000 7fb45d2fe380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T01:57:02.286 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.280+0000 7fb421fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T01:57:02.287 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.280+0000 7fb45d2fe380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T01:57:02.287 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:02.280+0000 7fb45d2fe380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T01:57:06.251 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.247+0000 7fb45d2fe380 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-17T01:57:06.251 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.247+0000 7fb45d2fe380 5 note: GC not initialized 2021-09-17T01:57:06.252 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.249+0000 7fb3827fc700 20 reqs_thread_entry: start 2021-09-17T01:57:06.252 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.249+0000 7fb3d8ff9700 20 reqs_thread_entry: start 2021-09-17T01:57:06.252 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.249+0000 7fb45d2fe380 20 init_complete bucket index max shards: 11 2021-09-17T01:57:06.252 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.249+0000 7fb45d2fe380 10 rgw notify: Started notification manager with: 1 workers 2021-09-17T01:57:06.253 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.249+0000 7fb380ff9700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 17 01:57:36 2021 2021-09-17T01:57:06.253 INFO:teuthology.orchestra.run.smithi013.stderr: 2021-09-17T01:57:06.253 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.249+0000 7fb36ffff700 20 reqs_thread_entry: start 2021-09-17T01:57:06.272 INFO:teuthology.orchestra.run.smithi013.stdout:[] 2021-09-17T01:57:06.272 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.270+0000 7fb45d2fe380 20 remove_watcher() i=0 2021-09-17T01:57:06.272 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.270+0000 7fb45d2fe380 2 removed watcher, disabling cache 2021-09-17T01:57:06.273 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.271+0000 7fb45d2fe380 20 remove_watcher() i=1 2021-09-17T01:57:06.274 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.272+0000 7fb45d2fe380 20 remove_watcher() i=2 2021-09-17T01:57:06.275 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.273+0000 7fb45d2fe380 20 remove_watcher() i=3 2021-09-17T01:57:06.276 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.275+0000 7fb45d2fe380 20 remove_watcher() i=4 2021-09-17T01:57:06.277 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.276+0000 7fb45d2fe380 20 remove_watcher() i=5 2021-09-17T01:57:06.278 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.277+0000 7fb45d2fe380 20 remove_watcher() i=6 2021-09-17T01:57:06.279 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.278+0000 7fb45d2fe380 20 remove_watcher() i=7 2021-09-17T01:57:06.280 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.279+0000 7fb421fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T01:57:06.286 DEBUG:tasks.util.rgw: json result: [] 2021-09-17T01:57:06.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T01:57:06.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T01:57:06.287 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-17T01:57:06.334 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:06.335 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:06.341 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.339+0000 7f3f75cb5380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T01:57:06.341 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.339+0000 7f3f75cb5380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T01:57:06.345 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.343+0000 7f3eb27fc700 20 reqs_thread_entry: start 2021-09-17T01:57:06.345 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.343+0000 7f3f75cb5380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-17T01:57:06.345 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.343+0000 7f3f75cb5380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.346 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.344+0000 7f3f75cb5380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:06.346 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.344+0000 7f3f75cb5380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.346 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.345+0000 7f3f75cb5380 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T01:57:06.355 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.353+0000 7f3f75cb5380 20 RGWRados::pool_iterate: got zone_info.ab0e682e-dc0c-48fc-b378-7171369d6437 2021-09-17T01:57:06.355 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.353+0000 7f3f75cb5380 20 RGWRados::pool_iterate: got zonegroup_info.da0d806e-dc17-459c-bcd4-045d23fda180 2021-09-17T01:57:06.355 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.353+0000 7f3f75cb5380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T01:57:06.355 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.353+0000 7f3f75cb5380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T01:57:06.356 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.353+0000 7f3f75cb5380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.356 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.353+0000 7f3f75cb5380 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T01:57:06.356 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.353+0000 7f3f75cb5380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.356 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.353+0000 7f3f75cb5380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.357 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.353+0000 7f3f75cb5380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.357 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.353+0000 7f3f75cb5380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:06.357 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.353+0000 7f3f75cb5380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.357 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.354+0000 7f3f75cb5380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:57:06.359 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:57:06.360 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "ab0e682e-dc0c-48fc-b378-7171369d6437", 2021-09-17T01:57:06.360 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "default", 2021-09-17T01:57:06.360 INFO:teuthology.orchestra.run.smithi013.stdout: "domain_root": "default.rgw.meta:root", 2021-09-17T01:57:06.360 INFO:teuthology.orchestra.run.smithi013.stdout: "control_pool": "default.rgw.control", 2021-09-17T01:57:06.360 INFO:teuthology.orchestra.run.smithi013.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-17T01:57:06.361 INFO:teuthology.orchestra.run.smithi013.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-17T01:57:06.361 INFO:teuthology.orchestra.run.smithi013.stdout: "log_pool": "default.rgw.log", 2021-09-17T01:57:06.361 INFO:teuthology.orchestra.run.smithi013.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-17T01:57:06.361 INFO:teuthology.orchestra.run.smithi013.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-17T01:57:06.362 INFO:teuthology.orchestra.run.smithi013.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-17T01:57:06.362 INFO:teuthology.orchestra.run.smithi013.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-17T01:57:06.362 INFO:teuthology.orchestra.run.smithi013.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-17T01:57:06.362 INFO:teuthology.orchestra.run.smithi013.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-17T01:57:06.362 INFO:teuthology.orchestra.run.smithi013.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-17T01:57:06.363 INFO:teuthology.orchestra.run.smithi013.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-17T01:57:06.363 INFO:teuthology.orchestra.run.smithi013.stdout: "otp_pool": "default.rgw.otp", 2021-09-17T01:57:06.363 INFO:teuthology.orchestra.run.smithi013.stdout: "system_key": { 2021-09-17T01:57:06.363 INFO:teuthology.orchestra.run.smithi013.stdout: "access_key": "", 2021-09-17T01:57:06.363 INFO:teuthology.orchestra.run.smithi013.stdout: "secret_key": "" 2021-09-17T01:57:06.364 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:06.364 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_pools": [ 2021-09-17T01:57:06.364 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:57:06.364 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "default-placement", 2021-09-17T01:57:06.365 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T01:57:06.365 INFO:teuthology.orchestra.run.smithi013.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-17T01:57:06.365 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": { 2021-09-17T01:57:06.365 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD": { 2021-09-17T01:57:06.365 INFO:teuthology.orchestra.run.smithi013.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-17T01:57:06.366 INFO:teuthology.orchestra.run.smithi013.stdout: "compression_type": "random" 2021-09-17T01:57:06.366 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:06.366 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:06.366 INFO:teuthology.orchestra.run.smithi013.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-17T01:57:06.366 INFO:teuthology.orchestra.run.smithi013.stdout: "index_type": 0 2021-09-17T01:57:06.367 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:06.367 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:06.367 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:06.367 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "", 2021-09-17T01:57:06.367 INFO:teuthology.orchestra.run.smithi013.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-17T01:57:06.368 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:57:06.369 DEBUG:tasks.util.rgw: json result: {'id': 'ab0e682e-dc0c-48fc-b378-7171369d6437', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-17T01:57:06.369 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-09-17T01:57:06.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-09-17T01:57:06.370 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-09-17T01:57:06.409 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:06.409 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:06.415 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.413+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.416 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.414+0000 7f8aa9035380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.416 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.414+0000 7f8aa9035380 20 realm 2021-09-17T01:57:06.416 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.414+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.417 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.414+0000 7f8aa9035380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.417 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.414+0000 7f8aa9035380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-17T01:57:06.417 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.414+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.417 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.414+0000 7f8aa9035380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.417 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.414+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.418 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.415+0000 7f8aa9035380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:06.418 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.415+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.419 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.416+0000 7f8aa9035380 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T01:57:06.419 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.416+0000 7f8aa9035380 20 searching for the correct realm 2021-09-17T01:57:06.426 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.423+0000 7f8aa9035380 20 RGWRados::pool_iterate: got zone_info.ab0e682e-dc0c-48fc-b378-7171369d6437 2021-09-17T01:57:06.426 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.423+0000 7f8aa9035380 20 RGWRados::pool_iterate: got zonegroup_info.da0d806e-dc17-459c-bcd4-045d23fda180 2021-09-17T01:57:06.426 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.423+0000 7f8aa9035380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T01:57:06.426 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.423+0000 7f8aa9035380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T01:57:06.427 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.423+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.427 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.423+0000 7f8aa9035380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.427 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.423+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.427 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.424+0000 7f8aa9035380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:06.428 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.424+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.428 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.424+0000 7f8aa9035380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:57:06.428 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.424+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.428 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.424+0000 7f8aa9035380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.428 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.424+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.429 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.424+0000 7f8aa9035380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.429 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.424+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.429 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.424+0000 7f8aa9035380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:06.433 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.431+0000 7f8aa9035380 20 RGWRados::pool_iterate: got zone_info.ab0e682e-dc0c-48fc-b378-7171369d6437 2021-09-17T01:57:06.434 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.431+0000 7f8aa9035380 20 RGWRados::pool_iterate: got zonegroup_info.da0d806e-dc17-459c-bcd4-045d23fda180 2021-09-17T01:57:06.434 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.431+0000 7f8aa9035380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T01:57:06.434 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.431+0000 7f8aa9035380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T01:57:06.434 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.431+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.435 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.431+0000 7f8aa9035380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:06.435 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.431+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.435 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T01:57:06.435 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.435 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:06.436 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.436 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:57:06.436 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.437 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.438 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 20 zone default found 2021-09-17T01:57:06.438 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 1 Realm: () 2021-09-17T01:57:06.438 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 1 ZoneGroup: default (da0d806e-dc17-459c-bcd4-045d23fda180) 2021-09-17T01:57:06.438 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 1 Zone: default (ab0e682e-dc0c-48fc-b378-7171369d6437) 2021-09-17T01:57:06.439 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-17T01:57:06.439 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 20 zonegroup default 2021-09-17T01:57:06.439 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.432+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.439 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.433+0000 7f8aa9035380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.439 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.433+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.440 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.433+0000 7f8aa9035380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.440 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.433+0000 7f8aa9035380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.440 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.433+0000 7f8aa9035380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.440 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.433+0000 7f8aa9035380 20 started sync module instance, tier type = 2021-09-17T01:57:06.440 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.433+0000 7f8aa9035380 20 started zone id=ab0e682e-dc0c-48fc-b378-7171369d6437 (name=default) with tier type = 2021-09-17T01:57:06.443 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.441+0000 7f8aa9035380 20 add_watcher() i=0 2021-09-17T01:57:06.443 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.441+0000 7f8aa9035380 20 add_watcher() i=1 2021-09-17T01:57:06.444 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.441+0000 7f8aa9035380 20 add_watcher() i=2 2021-09-17T01:57:06.444 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.441+0000 7f8aa9035380 20 add_watcher() i=3 2021-09-17T01:57:06.444 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.441+0000 7f8aa9035380 20 add_watcher() i=4 2021-09-17T01:57:06.444 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.441+0000 7f8aa9035380 20 add_watcher() i=5 2021-09-17T01:57:06.445 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.441+0000 7f8aa9035380 20 add_watcher() i=6 2021-09-17T01:57:06.445 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.442+0000 7f8aa9035380 20 add_watcher() i=7 2021-09-17T01:57:06.445 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.442+0000 7f8aa9035380 2 all 8 watchers are set, enabling cache 2021-09-17T01:57:06.446 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.443+0000 7f8a6dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T01:57:06.446 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.443+0000 7f8aa9035380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T01:57:06.446 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.443+0000 7f8aa9035380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T01:57:06.448 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.447+0000 7f8aa9035380 5 note: GC not initialized 2021-09-17T01:57:06.450 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.448+0000 7f89e4ff9700 20 reqs_thread_entry: start 2021-09-17T01:57:06.450 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.448+0000 7f89da7fc700 20 reqs_thread_entry: start 2021-09-17T01:57:06.450 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.448+0000 7f8aa9035380 20 init_complete bucket index max shards: 11 2021-09-17T01:57:06.450 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.448+0000 7f8aa9035380 10 rgw notify: Started notification manager with: 1 workers 2021-09-17T01:57:06.451 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.448+0000 7f89d8ff9700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 17 01:57:36 2021 2021-09-17T01:57:06.451 INFO:teuthology.orchestra.run.smithi013.stderr: 2021-09-17T01:57:06.451 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.448+0000 7f89bffff700 20 reqs_thread_entry: start 2021-09-17T01:57:06.457 INFO:teuthology.orchestra.run.smithi013.stdout:[] 2021-09-17T01:57:06.458 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.456+0000 7f8aa9035380 20 remove_watcher() i=0 2021-09-17T01:57:06.458 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.456+0000 7f8aa9035380 2 removed watcher, disabling cache 2021-09-17T01:57:06.459 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.457+0000 7f8aa9035380 20 remove_watcher() i=1 2021-09-17T01:57:06.459 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.458+0000 7f8aa9035380 20 remove_watcher() i=2 2021-09-17T01:57:06.460 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.458+0000 7f8aa9035380 20 remove_watcher() i=3 2021-09-17T01:57:06.461 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.459+0000 7f8aa9035380 20 remove_watcher() i=4 2021-09-17T01:57:06.461 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.460+0000 7f8aa9035380 20 remove_watcher() i=5 2021-09-17T01:57:06.462 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.461+0000 7f8aa9035380 20 remove_watcher() i=6 2021-09-17T01:57:06.463 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.462+0000 7f8aa9035380 20 remove_watcher() i=7 2021-09-17T01:57:06.464 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.462+0000 7f8a6dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T01:57:06.468 DEBUG:tasks.util.rgw: json result: [] 2021-09-17T01:57:06.469 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T01:57:06.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T01:57:06.470 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-17T01:57:06.552 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:06.552 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:06.558 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.556+0000 7f21d164b380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T01:57:06.558 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.556+0000 7f21d164b380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T01:57:06.561 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.560+0000 7f210dffb700 20 reqs_thread_entry: start 2021-09-17T01:57:06.562 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.560+0000 7f21d164b380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-17T01:57:06.562 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.560+0000 7f21d164b380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.563 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.561+0000 7f21d164b380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:06.563 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.561+0000 7f21d164b380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.563 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.561+0000 7f21d164b380 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T01:57:06.571 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.568+0000 7f21d164b380 20 RGWRados::pool_iterate: got zone_info.ab0e682e-dc0c-48fc-b378-7171369d6437 2021-09-17T01:57:06.571 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.568+0000 7f21d164b380 20 RGWRados::pool_iterate: got zonegroup_info.da0d806e-dc17-459c-bcd4-045d23fda180 2021-09-17T01:57:06.571 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.568+0000 7f21d164b380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T01:57:06.572 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.568+0000 7f21d164b380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T01:57:06.572 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.568+0000 7f21d164b380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.572 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.569+0000 7f21d164b380 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T01:57:06.572 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.569+0000 7f21d164b380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.572 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.569+0000 7f21d164b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:06.573 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.569+0000 7f21d164b380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.573 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.569+0000 7f21d164b380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:06.573 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.569+0000 7f21d164b380 20 rados->read ofs=0 len=0 2021-09-17T01:57:06.573 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:06.569+0000 7f21d164b380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:57:06.574 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:57:06.575 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "ab0e682e-dc0c-48fc-b378-7171369d6437", 2021-09-17T01:57:06.575 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "default", 2021-09-17T01:57:06.575 INFO:teuthology.orchestra.run.smithi013.stdout: "domain_root": "default.rgw.meta:root", 2021-09-17T01:57:06.575 INFO:teuthology.orchestra.run.smithi013.stdout: "control_pool": "default.rgw.control", 2021-09-17T01:57:06.576 INFO:teuthology.orchestra.run.smithi013.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-17T01:57:06.576 INFO:teuthology.orchestra.run.smithi013.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-17T01:57:06.576 INFO:teuthology.orchestra.run.smithi013.stdout: "log_pool": "default.rgw.log", 2021-09-17T01:57:06.576 INFO:teuthology.orchestra.run.smithi013.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-17T01:57:06.576 INFO:teuthology.orchestra.run.smithi013.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-17T01:57:06.577 INFO:teuthology.orchestra.run.smithi013.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-17T01:57:06.577 INFO:teuthology.orchestra.run.smithi013.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-17T01:57:06.577 INFO:teuthology.orchestra.run.smithi013.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-17T01:57:06.577 INFO:teuthology.orchestra.run.smithi013.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-17T01:57:06.577 INFO:teuthology.orchestra.run.smithi013.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-17T01:57:06.578 INFO:teuthology.orchestra.run.smithi013.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-17T01:57:06.578 INFO:teuthology.orchestra.run.smithi013.stdout: "otp_pool": "default.rgw.otp", 2021-09-17T01:57:06.578 INFO:teuthology.orchestra.run.smithi013.stdout: "system_key": { 2021-09-17T01:57:06.578 INFO:teuthology.orchestra.run.smithi013.stdout: "access_key": "", 2021-09-17T01:57:06.578 INFO:teuthology.orchestra.run.smithi013.stdout: "secret_key": "" 2021-09-17T01:57:06.579 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:06.579 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_pools": [ 2021-09-17T01:57:06.579 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:57:06.579 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "default-placement", 2021-09-17T01:57:06.579 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T01:57:06.580 INFO:teuthology.orchestra.run.smithi013.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-17T01:57:06.580 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": { 2021-09-17T01:57:06.580 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD": { 2021-09-17T01:57:06.580 INFO:teuthology.orchestra.run.smithi013.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-17T01:57:06.581 INFO:teuthology.orchestra.run.smithi013.stdout: "compression_type": "random" 2021-09-17T01:57:06.581 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:06.581 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:06.581 INFO:teuthology.orchestra.run.smithi013.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-17T01:57:06.581 INFO:teuthology.orchestra.run.smithi013.stdout: "index_type": 0 2021-09-17T01:57:06.582 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:06.582 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:06.582 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:06.582 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "", 2021-09-17T01:57:06.583 INFO:teuthology.orchestra.run.smithi013.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-17T01:57:06.583 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:57:06.584 DEBUG:tasks.util.rgw: json result: {'id': 'ab0e682e-dc0c-48fc-b378-7171369d6437', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-17T01:57:06.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-09-17T01:57:06.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-09-17T01:57:06.585 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-09-17T01:57:06.656 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:06.657 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:06.663 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:06.661+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.532 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.530+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.533 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.531+0000 7ffa87778380 20 realm 2021-09-17T01:57:08.533 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.531+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.536 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.534+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.536 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.534+0000 7ffa87778380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-17T01:57:08.536 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.534+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.537 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.534+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.538 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.534+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.538 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.535+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.538 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.535+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.535+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.535+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.535+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.535+0000 7ffa87778380 10 Creating default zonegroup 2021-09-17T01:57:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.535+0000 7ffa87778380 20 get_system_obj_state: rctx=0x7ffc3bc84320 obj=:.pools.avail state=0x55a1d72a95c0 s->prefetch_data=0 2021-09-17T01:57:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.536+0000 7ffa87778380 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-17T01:57:08.547 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.546+0000 7ffa87778380 10 WARNING: store->list_zones() returned r=-2 2021-09-17T01:57:08.548 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.546+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.548 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.546+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.550 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.548+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.550 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.548+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.551 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.548+0000 7ffa87778380 10 could not read realm id: (2) No such file or directory 2021-09-17T01:57:08.551 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.548+0000 7ffa87778380 10 WARNING: failed to set zone as default, r=-22 2021-09-17T01:57:08.551 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.548+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.551 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.549+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.553 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.551+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.551+0000 7ffa87778380 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T01:57:08.555 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.551+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.555 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.552+0000 7ffa87778380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:57:08.555 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.552+0000 7ffa87778380 10 Using default name default 2021-09-17T01:57:08.555 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.552+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.556 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.552+0000 7ffa87778380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:08.556 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.552+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.556 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.552+0000 7ffa87778380 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T01:57:08.556 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.552+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.556 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.553+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.557 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.553+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.557 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.553+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.557 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.553+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.558 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.553+0000 7ffa87778380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:08.563 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.561+0000 7ffa87778380 20 RGWRados::pool_iterate: got zone_info.5a668bc3-bc24-4f58-9608-769d0e80354a 2021-09-17T01:57:08.564 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.561+0000 7ffa87778380 20 RGWRados::pool_iterate: got zonegroup_info.64d55fd0-c5a2-4d37-bd67-59a9cd2f6255 2021-09-17T01:57:08.564 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.561+0000 7ffa87778380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T01:57:08.564 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.561+0000 7ffa87778380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T01:57:08.565 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.561+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.565 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.561+0000 7ffa87778380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:08.565 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.561+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.565 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T01:57:08.565 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.566 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:08.566 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.567 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:57:08.567 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.568 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.568 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 20 zone default found 2021-09-17T01:57:08.568 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 1 Realm: () 2021-09-17T01:57:08.568 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 1 ZoneGroup: default (64d55fd0-c5a2-4d37-bd67-59a9cd2f6255) 2021-09-17T01:57:08.569 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 1 Zone: default (5a668bc3-bc24-4f58-9608-769d0e80354a) 2021-09-17T01:57:08.569 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-17T01:57:08.569 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 20 zonegroup default 2021-09-17T01:57:08.569 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.562+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:08.569 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.563+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:08.570 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:08.563+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:10.541+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:10.544 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:10.542+0000 7ffa87778380 20 rados->read ofs=0 len=0 2021-09-17T01:57:10.545 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:10.542+0000 7ffa87778380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:10.545 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:10.542+0000 7ffa87778380 20 started sync module instance, tier type = 2021-09-17T01:57:10.545 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:10.542+0000 7ffa87778380 20 started zone id=5a668bc3-bc24-4f58-9608-769d0e80354a (name=default) with tier type = 2021-09-17T01:57:12.564 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.562+0000 7ffa87778380 20 add_watcher() i=0 2021-09-17T01:57:12.564 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.562+0000 7ffa87778380 20 add_watcher() i=1 2021-09-17T01:57:12.565 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.562+0000 7ffa87778380 20 add_watcher() i=2 2021-09-17T01:57:12.565 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.562+0000 7ffa87778380 20 add_watcher() i=3 2021-09-17T01:57:12.565 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.562+0000 7ffa87778380 20 add_watcher() i=4 2021-09-17T01:57:12.565 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.562+0000 7ffa87778380 20 add_watcher() i=5 2021-09-17T01:57:12.565 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.562+0000 7ffa87778380 20 add_watcher() i=6 2021-09-17T01:57:12.567 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.563+0000 7ffa87778380 20 add_watcher() i=7 2021-09-17T01:57:12.567 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.563+0000 7ffa87778380 2 all 8 watchers are set, enabling cache 2021-09-17T01:57:12.568 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.563+0000 7ffa87778380 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-17T01:57:12.568 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.563+0000 7ffa87778380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-17T01:57:12.568 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.563+0000 7ffa87778380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T01:57:12.568 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.564+0000 7ffa87778380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-17T01:57:12.569 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.564+0000 7ffa87778380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-17T01:57:12.569 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.564+0000 7ffa87778380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T01:57:12.569 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.564+0000 7ffa87778380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-17T01:57:12.569 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.564+0000 7ffa87778380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-17T01:57:12.570 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.564+0000 7ffa87778380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T01:57:12.570 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.564+0000 7ffa87778380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-17T01:57:12.570 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.564+0000 7ffa87778380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-17T01:57:12.570 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.564+0000 7ffa87778380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T01:57:12.570 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.564+0000 7ffa87778380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-17T01:57:12.571 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.564+0000 7ffa87778380 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-17T01:57:12.571 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.565+0000 7ffa87778380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T01:57:12.572 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.568+0000 7ffa4bfe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T01:57:12.572 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.568+0000 7ffa87778380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T01:57:12.572 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:12.568+0000 7ffa87778380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T01:57:16.547 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.545+0000 7ffa87778380 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-17T01:57:16.548 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.545+0000 7ffa87778380 5 note: GC not initialized 2021-09-17T01:57:16.550 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.548+0000 7ff9b0ff9700 20 reqs_thread_entry: start 2021-09-17T01:57:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.548+0000 7ffa87778380 20 init_complete bucket index max shards: 11 2021-09-17T01:57:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.548+0000 7ff9b2ffd700 20 reqs_thread_entry: start 2021-09-17T01:57:16.551 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.548+0000 7ffa87778380 10 rgw notify: Started notification manager with: 1 workers 2021-09-17T01:57:16.552 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.548+0000 7ff9aaffd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 17 01:57:46 2021 2021-09-17T01:57:16.552 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-09-17T01:57:16.553 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.548+0000 7ff9aa7fc700 20 reqs_thread_entry: start 2021-09-17T01:57:16.574 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2021-09-17T01:57:16.575 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.572+0000 7ffa87778380 20 remove_watcher() i=0 2021-09-17T01:57:16.575 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.572+0000 7ffa87778380 2 removed watcher, disabling cache 2021-09-17T01:57:16.575 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.573+0000 7ffa87778380 20 remove_watcher() i=1 2021-09-17T01:57:16.576 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.574+0000 7ffa87778380 20 remove_watcher() i=2 2021-09-17T01:57:16.577 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.575+0000 7ffa87778380 20 remove_watcher() i=3 2021-09-17T01:57:16.578 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.576+0000 7ffa87778380 20 remove_watcher() i=4 2021-09-17T01:57:16.579 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.577+0000 7ffa87778380 20 remove_watcher() i=5 2021-09-17T01:57:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.578+0000 7ffa87778380 20 remove_watcher() i=6 2021-09-17T01:57:16.580 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.579+0000 7ffa87778380 20 remove_watcher() i=7 2021-09-17T01:57:16.581 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.580+0000 7ffa4bfe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T01:57:16.587 DEBUG:tasks.util.rgw: json result: [] 2021-09-17T01:57:16.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T01:57:16.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T01:57:16.589 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-17T01:57:16.636 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:16.636 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:16.642 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.640+0000 7fde798a9380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T01:57:16.642 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.640+0000 7fde798a9380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T01:57:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.644+0000 7fddb67fc700 20 reqs_thread_entry: start 2021-09-17T01:57:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.644+0000 7fde798a9380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-17T01:57:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.644+0000 7fde798a9380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.645+0000 7fde798a9380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.645+0000 7fde798a9380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.648 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.646+0000 7fde798a9380 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T01:57:16.657 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.653+0000 7fde798a9380 20 RGWRados::pool_iterate: got zone_info.5a668bc3-bc24-4f58-9608-769d0e80354a 2021-09-17T01:57:16.657 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.653+0000 7fde798a9380 20 RGWRados::pool_iterate: got zonegroup_info.64d55fd0-c5a2-4d37-bd67-59a9cd2f6255 2021-09-17T01:57:16.657 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.653+0000 7fde798a9380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T01:57:16.658 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.653+0000 7fde798a9380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T01:57:16.658 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.653+0000 7fde798a9380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.659 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.654+0000 7fde798a9380 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T01:57:16.659 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.654+0000 7fde798a9380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.659 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.654+0000 7fde798a9380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.659 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.654+0000 7fde798a9380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.654+0000 7fde798a9380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.654+0000 7fde798a9380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.660 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.654+0000 7fde798a9380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:57:16.661 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:57:16.661 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "5a668bc3-bc24-4f58-9608-769d0e80354a", 2021-09-17T01:57:16.661 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "default", 2021-09-17T01:57:16.661 INFO:teuthology.orchestra.run.smithi045.stdout: "domain_root": "default.rgw.meta:root", 2021-09-17T01:57:16.662 INFO:teuthology.orchestra.run.smithi045.stdout: "control_pool": "default.rgw.control", 2021-09-17T01:57:16.662 INFO:teuthology.orchestra.run.smithi045.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-17T01:57:16.662 INFO:teuthology.orchestra.run.smithi045.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-17T01:57:16.662 INFO:teuthology.orchestra.run.smithi045.stdout: "log_pool": "default.rgw.log", 2021-09-17T01:57:16.663 INFO:teuthology.orchestra.run.smithi045.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-17T01:57:16.663 INFO:teuthology.orchestra.run.smithi045.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-17T01:57:16.663 INFO:teuthology.orchestra.run.smithi045.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-17T01:57:16.663 INFO:teuthology.orchestra.run.smithi045.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-17T01:57:16.663 INFO:teuthology.orchestra.run.smithi045.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-17T01:57:16.664 INFO:teuthology.orchestra.run.smithi045.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-17T01:57:16.664 INFO:teuthology.orchestra.run.smithi045.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-17T01:57:16.664 INFO:teuthology.orchestra.run.smithi045.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-17T01:57:16.664 INFO:teuthology.orchestra.run.smithi045.stdout: "otp_pool": "default.rgw.otp", 2021-09-17T01:57:16.665 INFO:teuthology.orchestra.run.smithi045.stdout: "system_key": { 2021-09-17T01:57:16.665 INFO:teuthology.orchestra.run.smithi045.stdout: "access_key": "", 2021-09-17T01:57:16.665 INFO:teuthology.orchestra.run.smithi045.stdout: "secret_key": "" 2021-09-17T01:57:16.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:16.666 INFO:teuthology.orchestra.run.smithi045.stdout: "placement_pools": [ 2021-09-17T01:57:16.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:57:16.666 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "default-placement", 2021-09-17T01:57:16.666 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:57:16.666 INFO:teuthology.orchestra.run.smithi045.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-17T01:57:16.667 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_classes": { 2021-09-17T01:57:16.667 INFO:teuthology.orchestra.run.smithi045.stdout: "STANDARD": { 2021-09-17T01:57:16.667 INFO:teuthology.orchestra.run.smithi045.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-17T01:57:16.667 INFO:teuthology.orchestra.run.smithi045.stdout: "compression_type": "random" 2021-09-17T01:57:16.667 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:16.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:16.668 INFO:teuthology.orchestra.run.smithi045.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-17T01:57:16.668 INFO:teuthology.orchestra.run.smithi045.stdout: "index_type": 0 2021-09-17T01:57:16.668 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:16.669 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:16.669 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:57:16.669 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "", 2021-09-17T01:57:16.669 INFO:teuthology.orchestra.run.smithi045.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-17T01:57:16.669 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:57:16.670 DEBUG:tasks.util.rgw: json result: {'id': '5a668bc3-bc24-4f58-9608-769d0e80354a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-17T01:57:16.671 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-09-17T01:57:16.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-09-17T01:57:16.672 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-09-17T01:57:16.711 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:16.712 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.715+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.722 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.716+0000 7f0ae0db1380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.716+0000 7f0ae0db1380 20 realm 2021-09-17T01:57:16.723 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.716+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.724 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.716+0000 7f0ae0db1380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.724 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.716+0000 7f0ae0db1380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-17T01:57:16.725 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.716+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.725 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.716+0000 7f0ae0db1380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.726 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.716+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.726 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.716+0000 7f0ae0db1380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:16.727 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.716+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.727 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.717+0000 7f0ae0db1380 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T01:57:16.728 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.717+0000 7f0ae0db1380 20 searching for the correct realm 2021-09-17T01:57:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.724+0000 7f0ae0db1380 20 RGWRados::pool_iterate: got zone_info.5a668bc3-bc24-4f58-9608-769d0e80354a 2021-09-17T01:57:16.732 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.724+0000 7f0ae0db1380 20 RGWRados::pool_iterate: got zonegroup_info.64d55fd0-c5a2-4d37-bd67-59a9cd2f6255 2021-09-17T01:57:16.732 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.724+0000 7f0ae0db1380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T01:57:16.733 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.724+0000 7f0ae0db1380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T01:57:16.733 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.724+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.734 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.725+0000 7f0ae0db1380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.734 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.725+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.735 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.725+0000 7f0ae0db1380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:16.735 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.725+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.736 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.725+0000 7f0ae0db1380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:57:16.736 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.725+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.737 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.725+0000 7f0ae0db1380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.737 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.725+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.738 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.725+0000 7f0ae0db1380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.738 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.725+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.739 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.726+0000 7f0ae0db1380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:16.746 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.732+0000 7f0ae0db1380 20 RGWRados::pool_iterate: got zone_info.5a668bc3-bc24-4f58-9608-769d0e80354a 2021-09-17T01:57:16.746 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.732+0000 7f0ae0db1380 20 RGWRados::pool_iterate: got zonegroup_info.64d55fd0-c5a2-4d37-bd67-59a9cd2f6255 2021-09-17T01:57:16.747 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.732+0000 7f0ae0db1380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T01:57:16.747 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.732+0000 7f0ae0db1380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T01:57:16.747 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.732+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.748 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.732+0000 7f0ae0db1380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:16.748 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.732+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.749 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.732+0000 7f0ae0db1380 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T01:57:16.749 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.732+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.750 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:16.750 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.751 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:57:16.751 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.752 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.752 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 20 zone default found 2021-09-17T01:57:16.753 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 1 Realm: () 2021-09-17T01:57:16.753 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 1 ZoneGroup: default (64d55fd0-c5a2-4d37-bd67-59a9cd2f6255) 2021-09-17T01:57:16.753 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 1 Zone: default (5a668bc3-bc24-4f58-9608-769d0e80354a) 2021-09-17T01:57:16.754 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-17T01:57:16.754 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 20 zonegroup default 2021-09-17T01:57:16.755 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.755 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.756 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.733+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.756 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.734+0000 7f0ae0db1380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.757 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.734+0000 7f0ae0db1380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.757 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.734+0000 7f0ae0db1380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.758 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.734+0000 7f0ae0db1380 20 started sync module instance, tier type = 2021-09-17T01:57:16.758 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.734+0000 7f0ae0db1380 20 started zone id=5a668bc3-bc24-4f58-9608-769d0e80354a (name=default) with tier type = 2021-09-17T01:57:16.759 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.741+0000 7f0ae0db1380 20 add_watcher() i=0 2021-09-17T01:57:16.759 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.741+0000 7f0ae0db1380 20 add_watcher() i=1 2021-09-17T01:57:16.760 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.741+0000 7f0ae0db1380 20 add_watcher() i=2 2021-09-17T01:57:16.760 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.741+0000 7f0ae0db1380 20 add_watcher() i=3 2021-09-17T01:57:16.760 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.741+0000 7f0ae0db1380 20 add_watcher() i=4 2021-09-17T01:57:16.761 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.741+0000 7f0ae0db1380 20 add_watcher() i=5 2021-09-17T01:57:16.761 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.741+0000 7f0ae0db1380 20 add_watcher() i=6 2021-09-17T01:57:16.762 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.742+0000 7f0ae0db1380 20 add_watcher() i=7 2021-09-17T01:57:16.762 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.742+0000 7f0ae0db1380 2 all 8 watchers are set, enabling cache 2021-09-17T01:57:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.743+0000 7f0aa57e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T01:57:16.763 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.743+0000 7f0ae0db1380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T01:57:16.764 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.743+0000 7f0ae0db1380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T01:57:16.766 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.747+0000 7f0ae0db1380 5 note: GC not initialized 2021-09-17T01:57:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.748+0000 7f0a13fff700 20 reqs_thread_entry: start 2021-09-17T01:57:16.767 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.748+0000 7f0a11ffb700 20 reqs_thread_entry: start 2021-09-17T01:57:16.768 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.748+0000 7f0ae0db1380 20 init_complete bucket index max shards: 11 2021-09-17T01:57:16.768 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.748+0000 7f0ae0db1380 10 rgw notify: Started notification manager with: 1 workers 2021-09-17T01:57:16.769 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.748+0000 7f09fffff700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 17 01:57:46 2021 2021-09-17T01:57:16.769 INFO:teuthology.orchestra.run.smithi045.stderr: 2021-09-17T01:57:16.770 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.749+0000 7f09ff7fe700 20 reqs_thread_entry: start 2021-09-17T01:57:16.770 INFO:teuthology.orchestra.run.smithi045.stdout:[] 2021-09-17T01:57:16.774 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.757+0000 7f0ae0db1380 20 remove_watcher() i=0 2021-09-17T01:57:16.774 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.757+0000 7f0ae0db1380 2 removed watcher, disabling cache 2021-09-17T01:57:16.775 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.757+0000 7f0ae0db1380 20 remove_watcher() i=1 2021-09-17T01:57:16.775 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.758+0000 7f0ae0db1380 20 remove_watcher() i=2 2021-09-17T01:57:16.776 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.759+0000 7f0ae0db1380 20 remove_watcher() i=3 2021-09-17T01:57:16.776 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.760+0000 7f0ae0db1380 20 remove_watcher() i=4 2021-09-17T01:57:16.777 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.761+0000 7f0ae0db1380 20 remove_watcher() i=5 2021-09-17T01:57:16.777 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.761+0000 7f0ae0db1380 20 remove_watcher() i=6 2021-09-17T01:57:16.778 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.762+0000 7f0ae0db1380 20 remove_watcher() i=7 2021-09-17T01:57:16.778 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.763+0000 7f0aa57e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T01:57:16.779 DEBUG:tasks.util.rgw: json result: [] 2021-09-17T01:57:16.779 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T01:57:16.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T01:57:16.781 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-17T01:57:16.858 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:16.858 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:16.864 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.862+0000 7f893df8d380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T01:57:16.864 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.862+0000 7f893df8d380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T01:57:16.867 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.866+0000 7f887a7fc700 20 reqs_thread_entry: start 2021-09-17T01:57:16.868 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.866+0000 7f893df8d380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-17T01:57:16.868 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.866+0000 7f893df8d380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.869 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.866+0000 7f893df8d380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:16.869 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.866+0000 7f893df8d380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.869 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.867+0000 7f893df8d380 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T01:57:16.877 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.874+0000 7f893df8d380 20 RGWRados::pool_iterate: got zone_info.5a668bc3-bc24-4f58-9608-769d0e80354a 2021-09-17T01:57:16.878 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.874+0000 7f893df8d380 20 RGWRados::pool_iterate: got zonegroup_info.64d55fd0-c5a2-4d37-bd67-59a9cd2f6255 2021-09-17T01:57:16.878 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.874+0000 7f893df8d380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T01:57:16.878 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.874+0000 7f893df8d380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T01:57:16.879 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.874+0000 7f893df8d380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.879 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.874+0000 7f893df8d380 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T01:57:16.879 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.874+0000 7f893df8d380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.879 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.875+0000 7f893df8d380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T01:57:16.880 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.875+0000 7f893df8d380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.880 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.875+0000 7f893df8d380 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T01:57:16.880 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.875+0000 7f893df8d380 20 rados->read ofs=0 len=0 2021-09-17T01:57:16.881 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:16.875+0000 7f893df8d380 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T01:57:16.882 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:57:16.882 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "5a668bc3-bc24-4f58-9608-769d0e80354a", 2021-09-17T01:57:16.883 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "default", 2021-09-17T01:57:16.883 INFO:teuthology.orchestra.run.smithi045.stdout: "domain_root": "default.rgw.meta:root", 2021-09-17T01:57:16.883 INFO:teuthology.orchestra.run.smithi045.stdout: "control_pool": "default.rgw.control", 2021-09-17T01:57:16.884 INFO:teuthology.orchestra.run.smithi045.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-17T01:57:16.884 INFO:teuthology.orchestra.run.smithi045.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-17T01:57:16.885 INFO:teuthology.orchestra.run.smithi045.stdout: "log_pool": "default.rgw.log", 2021-09-17T01:57:16.885 INFO:teuthology.orchestra.run.smithi045.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-17T01:57:16.885 INFO:teuthology.orchestra.run.smithi045.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-17T01:57:16.886 INFO:teuthology.orchestra.run.smithi045.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-17T01:57:16.886 INFO:teuthology.orchestra.run.smithi045.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-17T01:57:16.886 INFO:teuthology.orchestra.run.smithi045.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-17T01:57:16.887 INFO:teuthology.orchestra.run.smithi045.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-17T01:57:16.887 INFO:teuthology.orchestra.run.smithi045.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-17T01:57:16.887 INFO:teuthology.orchestra.run.smithi045.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-17T01:57:16.887 INFO:teuthology.orchestra.run.smithi045.stdout: "otp_pool": "default.rgw.otp", 2021-09-17T01:57:16.888 INFO:teuthology.orchestra.run.smithi045.stdout: "system_key": { 2021-09-17T01:57:16.888 INFO:teuthology.orchestra.run.smithi045.stdout: "access_key": "", 2021-09-17T01:57:16.888 INFO:teuthology.orchestra.run.smithi045.stdout: "secret_key": "" 2021-09-17T01:57:16.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:16.889 INFO:teuthology.orchestra.run.smithi045.stdout: "placement_pools": [ 2021-09-17T01:57:16.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:57:16.889 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "default-placement", 2021-09-17T01:57:16.890 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:57:16.890 INFO:teuthology.orchestra.run.smithi045.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-17T01:57:16.890 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_classes": { 2021-09-17T01:57:16.891 INFO:teuthology.orchestra.run.smithi045.stdout: "STANDARD": { 2021-09-17T01:57:16.891 INFO:teuthology.orchestra.run.smithi045.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-17T01:57:16.891 INFO:teuthology.orchestra.run.smithi045.stdout: "compression_type": "random" 2021-09-17T01:57:16.892 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:16.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:16.892 INFO:teuthology.orchestra.run.smithi045.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-17T01:57:16.892 INFO:teuthology.orchestra.run.smithi045.stdout: "index_type": 0 2021-09-17T01:57:16.893 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:16.893 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:16.893 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:57:16.894 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "", 2021-09-17T01:57:16.894 INFO:teuthology.orchestra.run.smithi045.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-17T01:57:16.894 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:57:16.895 DEBUG:tasks.util.rgw: json result: {'id': '5a668bc3-bc24-4f58-9608-769d0e80354a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-17T01:57:16.895 INFO:tasks.rgw:Starting rgw... 2021-09-17T01:57:16.896 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-09-17T01:57:16.896 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-17T01:57:16.897 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-17T01:57:16.898 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-17T01:57:16.902 INFO:tasks.rgw.c1.client.0:Started 2021-09-17T01:57:16.902 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-09-17T01:57:16.903 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-17T01:57:16.903 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-17T01:57:16.904 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-17T01:57:16.907 INFO:tasks.rgw.c1.client.1:Started 2021-09-17T01:57:16.907 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-09-17T01:57:16.908 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-17T01:57:16.908 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-17T01:57:16.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-17T01:57:16.925 INFO:tasks.rgw.c2.client.0:Started 2021-09-17T01:57:16.925 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-09-17T01:57:16.925 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-17T01:57:16.926 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-17T01:57:16.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-17T01:57:16.928 INFO:tasks.rgw.c2.client.1:Started 2021-09-17T01:57:16.929 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi013.front.sepia.ceph.com:8000/ 2021-09-17T01:57:16.929 DEBUG:teuthology.orchestra.run.smithi013:> curl http://smithi013.front.sepia.ceph.com:8000/ 2021-09-17T01:57:16.964 INFO:teuthology.orchestra.run.smithi013.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:57:16.964 INFO:teuthology.orchestra.run.smithi013.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:57:16.965 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-17T01:57:16.966 INFO:teuthology.orchestra.run.smithi013.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi013.front.sepia.ceph.com port 8000: Connection refused 2021-09-17T01:57:17.968 DEBUG:teuthology.orchestra.run.smithi013:> curl http://smithi013.front.sepia.ceph.com:8000/ 2021-09-17T01:57:17.989 INFO:teuthology.orchestra.run.smithi013.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:57:17.989 INFO:teuthology.orchestra.run.smithi013.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:57:17.991 INFO:teuthology.orchestra.run.smithi013.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-17T01:57:17.992 INFO:teuthology.orchestra.run.smithi013.stdout:anonymous 2021-09-17T01:57:17.993 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi013.front.sepia.ceph.com:8001/ 2021-09-17T01:57:17.993 DEBUG:teuthology.orchestra.run.smithi013:> curl http://smithi013.front.sepia.ceph.com:8001/ 2021-09-17T01:57:18.053 INFO:teuthology.orchestra.run.smithi013.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:57:18.053 INFO:teuthology.orchestra.run.smithi013.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:57:18.055 INFO:teuthology.orchestra.run.smithi013.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-17T01:57:18.056 INFO:teuthology.orchestra.run.smithi013.stdout:anonymous 2021-09-17T01:57:18.056 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi045.front.sepia.ceph.com:8000/ 2021-09-17T01:57:18.057 DEBUG:teuthology.orchestra.run.smithi045:> curl http://smithi045.front.sepia.ceph.com:8000/ 2021-09-17T01:57:18.100 INFO:teuthology.orchestra.run.smithi045.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:57:18.101 INFO:teuthology.orchestra.run.smithi045.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:57:18.102 INFO:teuthology.orchestra.run.smithi045.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-17T01:57:18.103 INFO:teuthology.orchestra.run.smithi045.stdout:anonymous 2021-09-17T01:57:18.103 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi045.front.sepia.ceph.com:8001/ 2021-09-17T01:57:18.104 DEBUG:teuthology.orchestra.run.smithi045:> curl http://smithi045.front.sepia.ceph.com:8001/ 2021-09-17T01:57:18.164 INFO:teuthology.orchestra.run.smithi045.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:57:18.164 INFO:teuthology.orchestra.run.smithi045.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:57:18.166 INFO:teuthology.orchestra.run.smithi045.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-17T01:57:18.166 INFO:teuthology.orchestra.run.smithi045.stdout:anonymous 2021-09-17T01:57:18.167 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-09-17T01:57:18.290 INFO:tasks.rgw_multisite:creating realm.. 2021-09-17T01:57:18.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:18.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:18.292 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:18.338 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:18.338 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:18.369 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:57:18.370 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:57:18.370 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-realm", 2021-09-17T01:57:18.370 INFO:teuthology.orchestra.run.smithi013.stdout: "current_period": "d9cd66ce-d16c-4226-9f7c-60489089c5d1", 2021-09-17T01:57:18.370 INFO:teuthology.orchestra.run.smithi013.stdout: "epoch": 1 2021-09-17T01:57:18.371 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:57:18.372 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-09-17T01:57:18.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:18.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:18.373 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:18.416 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:18.416 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:18.428 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:18.426+0000 7f5da911a380 -1 old format 2021-09-17T01:57:18.434 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:57:18.434 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "test-zonegroup", 2021-09-17T01:57:18.434 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zonegroup", 2021-09-17T01:57:18.435 INFO:teuthology.orchestra.run.smithi013.stdout: "api_name": "", 2021-09-17T01:57:18.435 INFO:teuthology.orchestra.run.smithi013.stdout: "is_master": "true", 2021-09-17T01:57:18.435 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T01:57:18.435 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T01:57:18.435 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:18.436 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames": [], 2021-09-17T01:57:18.436 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames_s3website": [], 2021-09-17T01:57:18.436 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zone": "", 2021-09-17T01:57:18.436 INFO:teuthology.orchestra.run.smithi013.stdout: "zones": [], 2021-09-17T01:57:18.437 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_targets": [], 2021-09-17T01:57:18.437 INFO:teuthology.orchestra.run.smithi013.stdout: "default_placement": "", 2021-09-17T01:57:18.437 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:57:18.437 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_policy": { 2021-09-17T01:57:18.437 INFO:teuthology.orchestra.run.smithi013.stdout: "groups": [] 2021-09-17T01:57:18.438 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:18.438 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:57:18.439 INFO:tasks.rgw_multisite:creating master zone.. 2021-09-17T01:57:18.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--endpoints', 'http://smithi013.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:18.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--endpoints', 'http://smithi013.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:18.440 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key HXCQHMJWFX35AU7N --secret B32eJkOEzpLbQQLCICRnQRj2OnvubCoS --endpoints http://smithi013.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:18.529 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:18.529 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:18.556 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:57:18.557 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:57:18.557 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone1", 2021-09-17T01:57:18.557 INFO:teuthology.orchestra.run.smithi013.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-17T01:57:18.557 INFO:teuthology.orchestra.run.smithi013.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-17T01:57:18.557 INFO:teuthology.orchestra.run.smithi013.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-17T01:57:18.558 INFO:teuthology.orchestra.run.smithi013.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-17T01:57:18.558 INFO:teuthology.orchestra.run.smithi013.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-17T01:57:18.558 INFO:teuthology.orchestra.run.smithi013.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-17T01:57:18.558 INFO:teuthology.orchestra.run.smithi013.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-17T01:57:18.559 INFO:teuthology.orchestra.run.smithi013.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-17T01:57:18.559 INFO:teuthology.orchestra.run.smithi013.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-17T01:57:18.559 INFO:teuthology.orchestra.run.smithi013.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-17T01:57:18.559 INFO:teuthology.orchestra.run.smithi013.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-17T01:57:18.560 INFO:teuthology.orchestra.run.smithi013.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-17T01:57:18.560 INFO:teuthology.orchestra.run.smithi013.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-17T01:57:18.560 INFO:teuthology.orchestra.run.smithi013.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-17T01:57:18.560 INFO:teuthology.orchestra.run.smithi013.stdout: "system_key": { 2021-09-17T01:57:18.560 INFO:teuthology.orchestra.run.smithi013.stdout: "access_key": "HXCQHMJWFX35AU7N", 2021-09-17T01:57:18.561 INFO:teuthology.orchestra.run.smithi013.stdout: "secret_key": "B32eJkOEzpLbQQLCICRnQRj2OnvubCoS" 2021-09-17T01:57:18.561 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:18.561 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_pools": [ 2021-09-17T01:57:18.561 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:57:18.562 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "default-placement", 2021-09-17T01:57:18.562 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T01:57:18.562 INFO:teuthology.orchestra.run.smithi013.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-17T01:57:18.562 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": { 2021-09-17T01:57:18.562 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD": { 2021-09-17T01:57:18.563 INFO:teuthology.orchestra.run.smithi013.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2021-09-17T01:57:18.563 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:18.563 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:18.563 INFO:teuthology.orchestra.run.smithi013.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-17T01:57:18.564 INFO:teuthology.orchestra.run.smithi013.stdout: "index_type": 0 2021-09-17T01:57:18.564 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:18.564 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:18.564 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:18.564 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:57:18.565 INFO:teuthology.orchestra.run.smithi013.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-17T01:57:18.565 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:57:18.566 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-09-17T01:57:19.074 INFO:teuthology.orchestra.run.smithi013.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-09-17T01:57:19.086 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-09-17T01:57:20.098 INFO:teuthology.orchestra.run.smithi013.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-09-17T01:57:20.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:20.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:20.113 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:20.162 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:20.162 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:20.187 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:57:20.187 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:57:20.187 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone1", 2021-09-17T01:57:20.187 INFO:teuthology.orchestra.run.smithi013.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-17T01:57:20.188 INFO:teuthology.orchestra.run.smithi013.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-17T01:57:20.188 INFO:teuthology.orchestra.run.smithi013.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-17T01:57:20.188 INFO:teuthology.orchestra.run.smithi013.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-17T01:57:20.188 INFO:teuthology.orchestra.run.smithi013.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-17T01:57:20.189 INFO:teuthology.orchestra.run.smithi013.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-17T01:57:20.189 INFO:teuthology.orchestra.run.smithi013.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-17T01:57:20.189 INFO:teuthology.orchestra.run.smithi013.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-17T01:57:20.189 INFO:teuthology.orchestra.run.smithi013.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-17T01:57:20.189 INFO:teuthology.orchestra.run.smithi013.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-17T01:57:20.190 INFO:teuthology.orchestra.run.smithi013.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-17T01:57:20.190 INFO:teuthology.orchestra.run.smithi013.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-17T01:57:20.190 INFO:teuthology.orchestra.run.smithi013.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-17T01:57:20.190 INFO:teuthology.orchestra.run.smithi013.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-17T01:57:20.190 INFO:teuthology.orchestra.run.smithi013.stdout: "system_key": { 2021-09-17T01:57:20.191 INFO:teuthology.orchestra.run.smithi013.stdout: "access_key": "HXCQHMJWFX35AU7N", 2021-09-17T01:57:20.191 INFO:teuthology.orchestra.run.smithi013.stdout: "secret_key": "B32eJkOEzpLbQQLCICRnQRj2OnvubCoS" 2021-09-17T01:57:20.191 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:20.191 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_pools": [ 2021-09-17T01:57:20.191 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:57:20.192 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "default-placement", 2021-09-17T01:57:20.192 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T01:57:20.192 INFO:teuthology.orchestra.run.smithi013.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-17T01:57:20.192 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": { 2021-09-17T01:57:20.192 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD": { 2021-09-17T01:57:20.193 INFO:teuthology.orchestra.run.smithi013.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2021-09-17T01:57:20.193 INFO:teuthology.orchestra.run.smithi013.stdout: "compression_type": "random" 2021-09-17T01:57:20.193 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:20.193 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:20.194 INFO:teuthology.orchestra.run.smithi013.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-17T01:57:20.194 INFO:teuthology.orchestra.run.smithi013.stdout: "index_type": 0 2021-09-17T01:57:20.194 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:20.194 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:20.194 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:20.195 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "", 2021-09-17T01:57:20.195 INFO:teuthology.orchestra.run.smithi013.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-17T01:57:20.195 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:57:20.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:20.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:20.197 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:20.239 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:20.240 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:20.249 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:20.248+0000 7fa8a18bd380 0 period (d9cd66ce-d16c-4226-9f7c-60489089c5d1 does not have zone 95b0d488-6c59-4a37-b3ed-d5fa65f27678 configured 2021-09-17T01:57:28.148 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:57:28.148 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:57:28.149 INFO:teuthology.orchestra.run.smithi013.stdout: "epoch": 1, 2021-09-17T01:57:28.149 INFO:teuthology.orchestra.run.smithi013.stdout: "predecessor_uuid": "d9cd66ce-d16c-4226-9f7c-60489089c5d1", 2021-09-17T01:57:28.149 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": [], 2021-09-17T01:57:28.149 INFO:teuthology.orchestra.run.smithi013.stdout: "period_map": { 2021-09-17T01:57:28.150 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:57:28.150 INFO:teuthology.orchestra.run.smithi013.stdout: "zonegroups": [ 2021-09-17T01:57:28.150 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:57:28.150 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "test-zonegroup", 2021-09-17T01:57:28.151 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zonegroup", 2021-09-17T01:57:28.151 INFO:teuthology.orchestra.run.smithi013.stdout: "api_name": "", 2021-09-17T01:57:28.151 INFO:teuthology.orchestra.run.smithi013.stdout: "is_master": "true", 2021-09-17T01:57:28.151 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T01:57:28.151 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T01:57:28.152 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:28.152 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames": [], 2021-09-17T01:57:28.152 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames_s3website": [], 2021-09-17T01:57:28.152 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:57:28.153 INFO:teuthology.orchestra.run.smithi013.stdout: "zones": [ 2021-09-17T01:57:28.153 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:57:28.153 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:57:28.153 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone1", 2021-09-17T01:57:28.154 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T01:57:28.154 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T01:57:28.154 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:28.154 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T01:57:28.155 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "false", 2021-09-17T01:57:28.155 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T01:57:28.155 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T01:57:28.155 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "", 2021-09-17T01:57:28.155 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "true", 2021-09-17T01:57:28.156 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [], 2021-09-17T01:57:28.156 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T01:57:28.156 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:28.156 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:28.156 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_targets": [ 2021-09-17T01:57:28.157 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:57:28.157 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "default-placement", 2021-09-17T01:57:28.157 INFO:teuthology.orchestra.run.smithi013.stdout: "tags": [], 2021-09-17T01:57:28.157 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": [ 2021-09-17T01:57:28.158 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD" 2021-09-17T01:57:28.158 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T01:57:28.158 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:28.158 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:28.158 INFO:teuthology.orchestra.run.smithi013.stdout: "default_placement": "default-placement", 2021-09-17T01:57:28.159 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:57:28.159 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_policy": { 2021-09-17T01:57:28.159 INFO:teuthology.orchestra.run.smithi013.stdout: "groups": [] 2021-09-17T01:57:28.159 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:28.160 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:28.160 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:28.160 INFO:teuthology.orchestra.run.smithi013.stdout: "short_zone_ids": [ 2021-09-17T01:57:28.160 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:57:28.161 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:57:28.161 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 803492335 2021-09-17T01:57:28.161 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:28.161 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T01:57:28.162 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:28.162 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T01:57:28.162 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:57:28.162 INFO:teuthology.orchestra.run.smithi013.stdout: "period_config": { 2021-09-17T01:57:28.162 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_quota": { 2021-09-17T01:57:28.163 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T01:57:28.163 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T01:57:28.163 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T01:57:28.163 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T01:57:28.164 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T01:57:28.164 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:28.164 INFO:teuthology.orchestra.run.smithi013.stdout: "user_quota": { 2021-09-17T01:57:28.164 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T01:57:28.164 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T01:57:28.165 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T01:57:28.165 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T01:57:28.165 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T01:57:28.166 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:28.166 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:28.166 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:57:28.166 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_name": "test-realm", 2021-09-17T01:57:28.166 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T01:57:28.167 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:57:28.168 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-09-17T01:57:28.168 INFO:teuthology.orchestra.run:waiting for 300 2021-09-17T01:57:28.210 INFO:tasks.rgw.c1.client.0.smithi013.stdout:2021-09-17T01:57:28.208+0000 7f0dd5e24700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 36553) UID: 0 2021-09-17T01:57:28.210 INFO:tasks.rgw.c1.client.0.smithi013.stdout:2021-09-17T01:57:28.208+0000 7f0de6cc5440 -1 shutting down 2021-09-17T01:57:30.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-17T01:57:34.271 INFO:tasks.rgw.c1.client.0:Stopped 2021-09-17T01:57:34.271 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-17T01:57:34.272 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-17T01:57:34.275 INFO:tasks.rgw.c1.client.0:Started 2021-09-17T01:57:34.275 DEBUG:teuthology.orchestra.run.smithi013:> curl http://smithi013.front.sepia.ceph.com:8000 2021-09-17T01:57:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-17T01:57:34.297 INFO:teuthology.orchestra.run.smithi013.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:57:34.297 INFO:teuthology.orchestra.run.smithi013.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:57:34.298 INFO:teuthology.orchestra.run.smithi013.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi013.front.sepia.ceph.com port 8000: Connection refused 2021-09-17T01:57:35.299 DEBUG:teuthology.orchestra.run.smithi013:> curl http://smithi013.front.sepia.ceph.com:8000 2021-09-17T01:57:35.320 INFO:teuthology.orchestra.run.smithi013.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:57:35.320 INFO:teuthology.orchestra.run.smithi013.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:57:35.322 INFO:teuthology.orchestra.run.smithi013.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-17T01:57:35.323 INFO:teuthology.orchestra.run.smithi013.stdout:anonymous 2021-09-17T01:57:35.323 INFO:tasks.rgw_multisite:creating admin user.. 2021-09-17T01:57:35.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:35.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:35.324 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key HXCQHMJWFX35AU7N --secret B32eJkOEzpLbQQLCICRnQRj2OnvubCoS --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:35.415 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:35.416 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:35.454 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:57:35.455 INFO:teuthology.orchestra.run.smithi013.stdout: "user_id": "realm-admin", 2021-09-17T01:57:35.455 INFO:teuthology.orchestra.run.smithi013.stdout: "display_name": "Realm Admin", 2021-09-17T01:57:35.455 INFO:teuthology.orchestra.run.smithi013.stdout: "email": "", 2021-09-17T01:57:35.456 INFO:teuthology.orchestra.run.smithi013.stdout: "suspended": 0, 2021-09-17T01:57:35.456 INFO:teuthology.orchestra.run.smithi013.stdout: "max_buckets": 1000, 2021-09-17T01:57:35.456 INFO:teuthology.orchestra.run.smithi013.stdout: "subusers": [], 2021-09-17T01:57:35.456 INFO:teuthology.orchestra.run.smithi013.stdout: "keys": [ 2021-09-17T01:57:35.457 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:57:35.457 INFO:teuthology.orchestra.run.smithi013.stdout: "user": "realm-admin", 2021-09-17T01:57:35.457 INFO:teuthology.orchestra.run.smithi013.stdout: "access_key": "HXCQHMJWFX35AU7N", 2021-09-17T01:57:35.457 INFO:teuthology.orchestra.run.smithi013.stdout: "secret_key": "B32eJkOEzpLbQQLCICRnQRj2OnvubCoS" 2021-09-17T01:57:35.457 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:35.458 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:35.458 INFO:teuthology.orchestra.run.smithi013.stdout: "swift_keys": [], 2021-09-17T01:57:35.458 INFO:teuthology.orchestra.run.smithi013.stdout: "caps": [], 2021-09-17T01:57:35.458 INFO:teuthology.orchestra.run.smithi013.stdout: "op_mask": "read, write, delete", 2021-09-17T01:57:35.458 INFO:teuthology.orchestra.run.smithi013.stdout: "system": "true", 2021-09-17T01:57:35.459 INFO:teuthology.orchestra.run.smithi013.stdout: "default_placement": "", 2021-09-17T01:57:35.459 INFO:teuthology.orchestra.run.smithi013.stdout: "default_storage_class": "", 2021-09-17T01:57:35.459 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_tags": [], 2021-09-17T01:57:35.459 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_quota": { 2021-09-17T01:57:35.460 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T01:57:35.460 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T01:57:35.460 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T01:57:35.460 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T01:57:35.460 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T01:57:35.461 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:35.461 INFO:teuthology.orchestra.run.smithi013.stdout: "user_quota": { 2021-09-17T01:57:35.461 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T01:57:35.461 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T01:57:35.462 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T01:57:35.462 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T01:57:35.462 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T01:57:35.462 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:35.462 INFO:teuthology.orchestra.run.smithi013.stdout: "temp_url_keys": [], 2021-09-17T01:57:35.463 INFO:teuthology.orchestra.run.smithi013.stdout: "type": "rgw", 2021-09-17T01:57:35.463 INFO:teuthology.orchestra.run.smithi013.stdout: "mfa_ids": [] 2021-09-17T01:57:35.463 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:57:35.463 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:57:35.466 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-17T01:57:35.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi013.front.sepia.ceph.com:8000', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:35.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi013.front.sepia.ceph.com:8000', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:35.467 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi013.front.sepia.ceph.com:8000 --access-key HXCQHMJWFX35AU7N --secret B32eJkOEzpLbQQLCICRnQRj2OnvubCoS --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:35.520 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:35.520 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:35.607 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:57:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:57:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-realm", 2021-09-17T01:57:35.608 INFO:teuthology.orchestra.run.smithi045.stdout: "current_period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:57:35.609 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 2 2021-09-17T01:57:35.609 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:57:35.611 INFO:tasks.rgw_multisite:creating zone.. 2021-09-17T01:57:35.611 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--endpoints', 'http://smithi045.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:35.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--endpoints', 'http://smithi045.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:35.612 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key HXCQHMJWFX35AU7N --secret B32eJkOEzpLbQQLCICRnQRj2OnvubCoS --endpoints http://smithi045.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:35.664 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:35.664 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:35.688 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:35.687+0000 7f85132e1380 0 failed reading obj info from .rgw.root:zone_info.95b0d488-6c59-4a37-b3ed-d5fa65f27678: (2) No such file or directory 2021-09-17T01:57:35.688 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:35.687+0000 7f85132e1380 0 WARNING: could not read zone params for zone id=95b0d488-6c59-4a37-b3ed-d5fa65f27678 name=test-zone1 2021-09-17T01:57:35.692 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:57:35.692 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T01:57:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone2", 2021-09-17T01:57:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-17T01:57:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-17T01:57:35.693 INFO:teuthology.orchestra.run.smithi045.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-17T01:57:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-17T01:57:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-17T01:57:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-17T01:57:35.694 INFO:teuthology.orchestra.run.smithi045.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-17T01:57:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-17T01:57:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-17T01:57:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-17T01:57:35.695 INFO:teuthology.orchestra.run.smithi045.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-17T01:57:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-17T01:57:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-17T01:57:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-17T01:57:35.696 INFO:teuthology.orchestra.run.smithi045.stdout: "system_key": { 2021-09-17T01:57:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "access_key": "HXCQHMJWFX35AU7N", 2021-09-17T01:57:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "secret_key": "B32eJkOEzpLbQQLCICRnQRj2OnvubCoS" 2021-09-17T01:57:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: "placement_pools": [ 2021-09-17T01:57:35.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:57:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "default-placement", 2021-09-17T01:57:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:57:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-17T01:57:35.698 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_classes": { 2021-09-17T01:57:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "STANDARD": { 2021-09-17T01:57:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2021-09-17T01:57:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:35.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-17T01:57:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: "index_type": 0 2021-09-17T01:57:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:35.700 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:57:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:57:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-17T01:57:35.701 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:57:35.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-09-17T01:57:36.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-09-17T01:57:36.385 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-09-17T01:57:36.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-09-17T01:57:37.394 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-09-17T01:57:37.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:37.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:37.419 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:37.467 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:37.467 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:37.493 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:57:37.493 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T01:57:37.493 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone2", 2021-09-17T01:57:37.493 INFO:teuthology.orchestra.run.smithi045.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-17T01:57:37.494 INFO:teuthology.orchestra.run.smithi045.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-17T01:57:37.494 INFO:teuthology.orchestra.run.smithi045.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-17T01:57:37.494 INFO:teuthology.orchestra.run.smithi045.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-17T01:57:37.494 INFO:teuthology.orchestra.run.smithi045.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-17T01:57:37.495 INFO:teuthology.orchestra.run.smithi045.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-17T01:57:37.495 INFO:teuthology.orchestra.run.smithi045.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-17T01:57:37.495 INFO:teuthology.orchestra.run.smithi045.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-17T01:57:37.495 INFO:teuthology.orchestra.run.smithi045.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-17T01:57:37.496 INFO:teuthology.orchestra.run.smithi045.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-17T01:57:37.496 INFO:teuthology.orchestra.run.smithi045.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-17T01:57:37.496 INFO:teuthology.orchestra.run.smithi045.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-17T01:57:37.496 INFO:teuthology.orchestra.run.smithi045.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-17T01:57:37.496 INFO:teuthology.orchestra.run.smithi045.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-17T01:57:37.497 INFO:teuthology.orchestra.run.smithi045.stdout: "system_key": { 2021-09-17T01:57:37.497 INFO:teuthology.orchestra.run.smithi045.stdout: "access_key": "HXCQHMJWFX35AU7N", 2021-09-17T01:57:37.497 INFO:teuthology.orchestra.run.smithi045.stdout: "secret_key": "B32eJkOEzpLbQQLCICRnQRj2OnvubCoS" 2021-09-17T01:57:37.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:37.498 INFO:teuthology.orchestra.run.smithi045.stdout: "placement_pools": [ 2021-09-17T01:57:37.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:57:37.498 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "default-placement", 2021-09-17T01:57:37.498 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:57:37.498 INFO:teuthology.orchestra.run.smithi045.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-17T01:57:37.499 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_classes": { 2021-09-17T01:57:37.499 INFO:teuthology.orchestra.run.smithi045.stdout: "STANDARD": { 2021-09-17T01:57:37.499 INFO:teuthology.orchestra.run.smithi045.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-17T01:57:37.499 INFO:teuthology.orchestra.run.smithi045.stdout: "compression_type": "random" 2021-09-17T01:57:37.500 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:37.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:37.500 INFO:teuthology.orchestra.run.smithi045.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-17T01:57:37.500 INFO:teuthology.orchestra.run.smithi045.stdout: "index_type": 0 2021-09-17T01:57:37.501 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:37.501 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:37.501 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:57:37.501 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "", 2021-09-17T01:57:37.501 INFO:teuthology.orchestra.run.smithi045.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-17T01:57:37.502 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:57:37.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:37.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:37.504 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:37.546 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:37.547 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:37.555 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:57:37.555+0000 7fae6eaef380 0 period (3c964c05-2a9c-48af-9858-975bf7b57c1a does not have zone 54e01141-a1e3-4035-9545-e42ec73c8edc configured 2021-09-17T01:57:45.440 INFO:teuthology.orchestra.run.smithi045.stderr:Sending period to new master zone 95b0d488-6c59-4a37-b3ed-d5fa65f27678 2021-09-17T01:57:45.478 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:57:45.478 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:57:45.478 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 2, 2021-09-17T01:57:45.479 INFO:teuthology.orchestra.run.smithi045.stdout: "predecessor_uuid": "d9cd66ce-d16c-4226-9f7c-60489089c5d1", 2021-09-17T01:57:45.479 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": [], 2021-09-17T01:57:45.479 INFO:teuthology.orchestra.run.smithi045.stdout: "period_map": { 2021-09-17T01:57:45.479 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:57:45.480 INFO:teuthology.orchestra.run.smithi045.stdout: "zonegroups": [ 2021-09-17T01:57:45.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:57:45.480 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "test-zonegroup", 2021-09-17T01:57:45.480 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zonegroup", 2021-09-17T01:57:45.481 INFO:teuthology.orchestra.run.smithi045.stdout: "api_name": "", 2021-09-17T01:57:45.481 INFO:teuthology.orchestra.run.smithi045.stdout: "is_master": "true", 2021-09-17T01:57:45.481 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T01:57:45.481 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T01:57:45.481 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:57:45.482 INFO:teuthology.orchestra.run.smithi045.stdout: "hostnames": [], 2021-09-17T01:57:45.482 INFO:teuthology.orchestra.run.smithi045.stdout: "hostnames_s3website": [], 2021-09-17T01:57:45.482 INFO:teuthology.orchestra.run.smithi045.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:57:45.482 INFO:teuthology.orchestra.run.smithi045.stdout: "zones": [ 2021-09-17T01:57:45.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:57:45.483 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T01:57:45.483 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone2", 2021-09-17T01:57:45.483 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T01:57:45.483 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi045.front.sepia.ceph.com:8000" 2021-09-17T01:57:45.484 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:57:45.484 INFO:teuthology.orchestra.run.smithi045.stdout: "log_meta": "false", 2021-09-17T01:57:45.484 INFO:teuthology.orchestra.run.smithi045.stdout: "log_data": "true", 2021-09-17T01:57:45.484 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_index_max_shards": 11, 2021-09-17T01:57:45.485 INFO:teuthology.orchestra.run.smithi045.stdout: "read_only": "false", 2021-09-17T01:57:45.485 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_type": "", 2021-09-17T01:57:45.485 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from_all": "true", 2021-09-17T01:57:45.485 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from": [], 2021-09-17T01:57:45.485 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_zone": "" 2021-09-17T01:57:45.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:45.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:57:45.486 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:57:45.486 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone1", 2021-09-17T01:57:45.487 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T01:57:45.487 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T01:57:45.487 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:57:45.487 INFO:teuthology.orchestra.run.smithi045.stdout: "log_meta": "false", 2021-09-17T01:57:45.487 INFO:teuthology.orchestra.run.smithi045.stdout: "log_data": "true", 2021-09-17T01:57:45.488 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_index_max_shards": 11, 2021-09-17T01:57:45.488 INFO:teuthology.orchestra.run.smithi045.stdout: "read_only": "false", 2021-09-17T01:57:45.488 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_type": "", 2021-09-17T01:57:45.488 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from_all": "true", 2021-09-17T01:57:45.489 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from": [], 2021-09-17T01:57:45.489 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_zone": "" 2021-09-17T01:57:45.489 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:45.489 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:57:45.489 INFO:teuthology.orchestra.run.smithi045.stdout: "placement_targets": [ 2021-09-17T01:57:45.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:57:45.490 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "default-placement", 2021-09-17T01:57:45.490 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2021-09-17T01:57:45.490 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_classes": [ 2021-09-17T01:57:45.491 INFO:teuthology.orchestra.run.smithi045.stdout: "STANDARD" 2021-09-17T01:57:45.491 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:57:45.491 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:45.491 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:57:45.491 INFO:teuthology.orchestra.run.smithi045.stdout: "default_placement": "default-placement", 2021-09-17T01:57:45.492 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:57:45.492 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_policy": { 2021-09-17T01:57:45.492 INFO:teuthology.orchestra.run.smithi045.stdout: "groups": [] 2021-09-17T01:57:45.492 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:45.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:45.493 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:57:45.493 INFO:teuthology.orchestra.run.smithi045.stdout: "short_zone_ids": [ 2021-09-17T01:57:45.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:57:45.493 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T01:57:45.494 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 4091795418 2021-09-17T01:57:45.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:45.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:57:45.494 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:57:45.495 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 803492335 2021-09-17T01:57:45.495 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:45.495 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:57:45.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:45.495 INFO:teuthology.orchestra.run.smithi045.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T01:57:45.496 INFO:teuthology.orchestra.run.smithi045.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:57:45.496 INFO:teuthology.orchestra.run.smithi045.stdout: "period_config": { 2021-09-17T01:57:45.496 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_quota": { 2021-09-17T01:57:45.497 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": false, 2021-09-17T01:57:45.497 INFO:teuthology.orchestra.run.smithi045.stdout: "check_on_raw": false, 2021-09-17T01:57:45.497 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size": -1, 2021-09-17T01:57:45.497 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size_kb": 0, 2021-09-17T01:57:45.497 INFO:teuthology.orchestra.run.smithi045.stdout: "max_objects": -1 2021-09-17T01:57:45.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:45.498 INFO:teuthology.orchestra.run.smithi045.stdout: "user_quota": { 2021-09-17T01:57:45.498 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": false, 2021-09-17T01:57:45.498 INFO:teuthology.orchestra.run.smithi045.stdout: "check_on_raw": false, 2021-09-17T01:57:45.499 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size": -1, 2021-09-17T01:57:45.499 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size_kb": 0, 2021-09-17T01:57:45.499 INFO:teuthology.orchestra.run.smithi045.stdout: "max_objects": -1 2021-09-17T01:57:45.499 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:57:45.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:57:45.500 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:57:45.500 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_name": "test-realm", 2021-09-17T01:57:45.500 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:57:45.500 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:57:45.501 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-09-17T01:57:45.502 INFO:teuthology.orchestra.run:waiting for 300 2021-09-17T01:57:45.543 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T01:57:45.542+0000 7f112f61a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 36481) UID: 0 2021-09-17T01:57:45.543 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T01:57:45.542+0000 7f11404bb440 -1 shutting down 2021-09-17T01:57:46.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-09-17T01:57:51.604 INFO:tasks.rgw.c2.client.0:Stopped 2021-09-17T01:57:51.604 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-17T01:57:51.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-17T01:57:51.608 INFO:tasks.rgw.c2.client.0:Started 2021-09-17T01:57:51.608 DEBUG:teuthology.orchestra.run.smithi045:> curl http://smithi045.front.sepia.ceph.com:8000 2021-09-17T01:57:51.629 INFO:teuthology.orchestra.run.smithi045.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:57:51.630 INFO:teuthology.orchestra.run.smithi045.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:57:51.631 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-17T01:57:51.631 INFO:teuthology.orchestra.run.smithi045.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi045.front.sepia.ceph.com port 8000: Connection refused 2021-09-17T01:57:51.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2021-09-17T01:57:52.632 DEBUG:teuthology.orchestra.run.smithi045:> curl http://smithi045.front.sepia.ceph.com:8000 2021-09-17T01:57:52.656 INFO:teuthology.orchestra.run.smithi045.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:57:52.656 INFO:teuthology.orchestra.run.smithi045.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:57:52.657 INFO:teuthology.orchestra.run.smithi045.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-17T01:57:52.658 INFO:teuthology.orchestra.run.smithi045.stdout:anonymous 2021-09-17T01:57:52.659 INFO:tasks.rgw_multisite:creating zone.. 2021-09-17T01:57:52.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--endpoints', 'http://smithi013.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:52.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--endpoints', 'http://smithi013.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:52.660 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HXCQHMJWFX35AU7N --secret B32eJkOEzpLbQQLCICRnQRj2OnvubCoS --endpoints http://smithi013.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:52.715 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:52.715 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:52.743 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:52.742+0000 7f87bf334380 0 failed reading obj info from .rgw.root:zone_info.54e01141-a1e3-4035-9545-e42ec73c8edc: (2) No such file or directory 2021-09-17T01:57:52.743 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:52.742+0000 7f87bf334380 0 WARNING: could not read zone params for zone id=54e01141-a1e3-4035-9545-e42ec73c8edc name=test-zone2 2021-09-17T01:57:52.747 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:57:52.747 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T01:57:52.747 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone3", 2021-09-17T01:57:52.748 INFO:teuthology.orchestra.run.smithi013.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-17T01:57:52.748 INFO:teuthology.orchestra.run.smithi013.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-17T01:57:52.748 INFO:teuthology.orchestra.run.smithi013.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-17T01:57:52.748 INFO:teuthology.orchestra.run.smithi013.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-17T01:57:52.749 INFO:teuthology.orchestra.run.smithi013.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-17T01:57:52.749 INFO:teuthology.orchestra.run.smithi013.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-17T01:57:52.749 INFO:teuthology.orchestra.run.smithi013.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-17T01:57:52.749 INFO:teuthology.orchestra.run.smithi013.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-17T01:57:52.750 INFO:teuthology.orchestra.run.smithi013.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-17T01:57:52.750 INFO:teuthology.orchestra.run.smithi013.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-17T01:57:52.750 INFO:teuthology.orchestra.run.smithi013.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-17T01:57:52.750 INFO:teuthology.orchestra.run.smithi013.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-17T01:57:52.751 INFO:teuthology.orchestra.run.smithi013.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-17T01:57:52.751 INFO:teuthology.orchestra.run.smithi013.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-17T01:57:52.751 INFO:teuthology.orchestra.run.smithi013.stdout: "system_key": { 2021-09-17T01:57:52.751 INFO:teuthology.orchestra.run.smithi013.stdout: "access_key": "HXCQHMJWFX35AU7N", 2021-09-17T01:57:52.751 INFO:teuthology.orchestra.run.smithi013.stdout: "secret_key": "B32eJkOEzpLbQQLCICRnQRj2OnvubCoS" 2021-09-17T01:57:52.752 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:52.752 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_pools": [ 2021-09-17T01:57:52.752 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:57:52.752 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "default-placement", 2021-09-17T01:57:52.753 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T01:57:52.753 INFO:teuthology.orchestra.run.smithi013.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-17T01:57:52.753 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": { 2021-09-17T01:57:52.753 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD": { 2021-09-17T01:57:52.753 INFO:teuthology.orchestra.run.smithi013.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2021-09-17T01:57:52.754 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:52.754 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:52.754 INFO:teuthology.orchestra.run.smithi013.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-17T01:57:52.755 INFO:teuthology.orchestra.run.smithi013.stdout: "index_type": 0 2021-09-17T01:57:52.755 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:52.755 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:52.755 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:52.755 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:57:52.756 INFO:teuthology.orchestra.run.smithi013.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-17T01:57:52.756 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:57:52.757 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-09-17T01:57:53.278 INFO:teuthology.orchestra.run.smithi013.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-09-17T01:57:53.295 DEBUG:teuthology.orchestra.run.smithi013:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-09-17T01:57:54.291 INFO:teuthology.orchestra.run.smithi013.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-09-17T01:57:54.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:54.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:54.304 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:54.351 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:54.351 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:54.374 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:57:54.375 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T01:57:54.375 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone3", 2021-09-17T01:57:54.375 INFO:teuthology.orchestra.run.smithi013.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-17T01:57:54.375 INFO:teuthology.orchestra.run.smithi013.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-17T01:57:54.376 INFO:teuthology.orchestra.run.smithi013.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-17T01:57:54.376 INFO:teuthology.orchestra.run.smithi013.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-17T01:57:54.376 INFO:teuthology.orchestra.run.smithi013.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-17T01:57:54.377 INFO:teuthology.orchestra.run.smithi013.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-17T01:57:54.377 INFO:teuthology.orchestra.run.smithi013.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-17T01:57:54.377 INFO:teuthology.orchestra.run.smithi013.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-17T01:57:54.377 INFO:teuthology.orchestra.run.smithi013.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-17T01:57:54.378 INFO:teuthology.orchestra.run.smithi013.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-17T01:57:54.378 INFO:teuthology.orchestra.run.smithi013.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-17T01:57:54.378 INFO:teuthology.orchestra.run.smithi013.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-17T01:57:54.378 INFO:teuthology.orchestra.run.smithi013.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-17T01:57:54.378 INFO:teuthology.orchestra.run.smithi013.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-17T01:57:54.379 INFO:teuthology.orchestra.run.smithi013.stdout: "system_key": { 2021-09-17T01:57:54.379 INFO:teuthology.orchestra.run.smithi013.stdout: "access_key": "HXCQHMJWFX35AU7N", 2021-09-17T01:57:54.379 INFO:teuthology.orchestra.run.smithi013.stdout: "secret_key": "B32eJkOEzpLbQQLCICRnQRj2OnvubCoS" 2021-09-17T01:57:54.379 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:54.380 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_pools": [ 2021-09-17T01:57:54.380 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:57:54.380 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "default-placement", 2021-09-17T01:57:54.380 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T01:57:54.381 INFO:teuthology.orchestra.run.smithi013.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-17T01:57:54.381 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": { 2021-09-17T01:57:54.381 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD": { 2021-09-17T01:57:54.381 INFO:teuthology.orchestra.run.smithi013.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2021-09-17T01:57:54.382 INFO:teuthology.orchestra.run.smithi013.stdout: "compression_type": "random" 2021-09-17T01:57:54.382 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:54.382 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:57:54.382 INFO:teuthology.orchestra.run.smithi013.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-17T01:57:54.382 INFO:teuthology.orchestra.run.smithi013.stdout: "index_type": 0 2021-09-17T01:57:54.383 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:54.383 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:57:54.383 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:57:54.383 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "", 2021-09-17T01:57:54.384 INFO:teuthology.orchestra.run.smithi013.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-17T01:57:54.384 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:57:54.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:54.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:57:54.386 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T01:57:54.429 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:57:54.430 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:57:54.438 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T01:57:54.438+0000 7f9115a56380 0 period (3c964c05-2a9c-48af-9858-975bf7b57c1a does not have zone 505278ea-3c6d-4c68-8442-578099047a49 configured 2021-09-17T01:57:57.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-09-17T01:58:02.320 INFO:teuthology.orchestra.run.smithi013.stderr:Sending period to new master zone 95b0d488-6c59-4a37-b3ed-d5fa65f27678 2021-09-17T01:58:02.344 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:58:02.344 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:02.344 INFO:teuthology.orchestra.run.smithi013.stdout: "epoch": 3, 2021-09-17T01:58:02.345 INFO:teuthology.orchestra.run.smithi013.stdout: "predecessor_uuid": "d9cd66ce-d16c-4226-9f7c-60489089c5d1", 2021-09-17T01:58:02.345 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": [], 2021-09-17T01:58:02.345 INFO:teuthology.orchestra.run.smithi013.stdout: "period_map": { 2021-09-17T01:58:02.345 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:02.346 INFO:teuthology.orchestra.run.smithi013.stdout: "zonegroups": [ 2021-09-17T01:58:02.346 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:02.346 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "test-zonegroup", 2021-09-17T01:58:02.346 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zonegroup", 2021-09-17T01:58:02.347 INFO:teuthology.orchestra.run.smithi013.stdout: "api_name": "", 2021-09-17T01:58:02.347 INFO:teuthology.orchestra.run.smithi013.stdout: "is_master": "true", 2021-09-17T01:58:02.347 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T01:58:02.347 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T01:58:02.347 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:58:02.348 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames": [], 2021-09-17T01:58:02.348 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames_s3website": [], 2021-09-17T01:58:02.348 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:58:02.348 INFO:teuthology.orchestra.run.smithi013.stdout: "zones": [ 2021-09-17T01:58:02.349 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:02.349 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T01:58:02.349 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone3", 2021-09-17T01:58:02.349 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T01:58:02.349 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8001" 2021-09-17T01:58:02.350 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:58:02.350 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T01:58:02.350 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "true", 2021-09-17T01:58:02.350 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T01:58:02.351 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T01:58:02.351 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "", 2021-09-17T01:58:02.351 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "true", 2021-09-17T01:58:02.351 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [], 2021-09-17T01:58:02.351 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T01:58:02.352 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:02.352 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:02.352 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T01:58:02.352 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone2", 2021-09-17T01:58:02.353 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T01:58:02.353 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi045.front.sepia.ceph.com:8000" 2021-09-17T01:58:02.353 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:58:02.353 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T01:58:02.354 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "true", 2021-09-17T01:58:02.354 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T01:58:02.354 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T01:58:02.354 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "", 2021-09-17T01:58:02.354 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "true", 2021-09-17T01:58:02.355 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [], 2021-09-17T01:58:02.355 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T01:58:02.355 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:02.355 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:02.356 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:58:02.356 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone1", 2021-09-17T01:58:02.356 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T01:58:02.356 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T01:58:02.357 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:58:02.357 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T01:58:02.357 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "true", 2021-09-17T01:58:02.357 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T01:58:02.358 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T01:58:02.358 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "", 2021-09-17T01:58:02.358 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "true", 2021-09-17T01:58:02.358 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [], 2021-09-17T01:58:02.358 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T01:58:02.359 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:58:02.359 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:58:02.359 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_targets": [ 2021-09-17T01:58:02.359 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:02.359 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "default-placement", 2021-09-17T01:58:02.360 INFO:teuthology.orchestra.run.smithi013.stdout: "tags": [], 2021-09-17T01:58:02.360 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": [ 2021-09-17T01:58:02.360 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD" 2021-09-17T01:58:02.360 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T01:58:02.361 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:58:02.361 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:58:02.361 INFO:teuthology.orchestra.run.smithi013.stdout: "default_placement": "default-placement", 2021-09-17T01:58:02.361 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:58:02.361 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_policy": { 2021-09-17T01:58:02.362 INFO:teuthology.orchestra.run.smithi013.stdout: "groups": [] 2021-09-17T01:58:02.362 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:58:02.362 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:58:02.362 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:58:02.363 INFO:teuthology.orchestra.run.smithi013.stdout: "short_zone_ids": [ 2021-09-17T01:58:02.363 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:02.363 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T01:58:02.363 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 4005434042 2021-09-17T01:58:02.363 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:02.364 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:02.364 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T01:58:02.364 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 4091795418 2021-09-17T01:58:02.364 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:02.365 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:02.365 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:58:02.365 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 803492335 2021-09-17T01:58:02.365 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:58:02.365 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T01:58:02.366 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:02.366 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T01:58:02.366 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:58:02.366 INFO:teuthology.orchestra.run.smithi013.stdout: "period_config": { 2021-09-17T01:58:02.366 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_quota": { 2021-09-17T01:58:02.367 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T01:58:02.367 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T01:58:02.367 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T01:58:02.367 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T01:58:02.368 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T01:58:02.368 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:02.368 INFO:teuthology.orchestra.run.smithi013.stdout: "user_quota": { 2021-09-17T01:58:02.368 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T01:58:02.368 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T01:58:02.370 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T01:58:02.370 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T01:58:02.370 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T01:58:02.370 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:58:02.371 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:02.371 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:58:02.371 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_name": "test-realm", 2021-09-17T01:58:02.371 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T01:58:02.372 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:58:02.372 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-09-17T01:58:02.373 INFO:teuthology.orchestra.run:waiting for 300 2021-09-17T01:58:02.412 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T01:58:02.412+0000 7f7bf277c700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 36557) UID: 0 2021-09-17T01:58:02.413 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T01:58:02.412+0000 7f7c0361d440 -1 shutting down 2021-09-17T01:58:04.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-09-17T01:58:08.475 INFO:tasks.rgw.c1.client.1:Stopped 2021-09-17T01:58:08.475 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-17T01:58:08.476 DEBUG:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-17T01:58:08.479 INFO:tasks.rgw.c1.client.1:Started 2021-09-17T01:58:08.479 DEBUG:teuthology.orchestra.run.smithi013:> curl http://smithi013.front.sepia.ceph.com:8001 2021-09-17T01:58:08.502 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-17T01:58:08.502 INFO:teuthology.orchestra.run.smithi013.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:58:08.503 INFO:teuthology.orchestra.run.smithi013.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:58:08.503 INFO:teuthology.orchestra.run.smithi013.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi013.front.sepia.ceph.com port 8001: Connection refused 2021-09-17T01:58:09.504 DEBUG:teuthology.orchestra.run.smithi013:> curl http://smithi013.front.sepia.ceph.com:8001 2021-09-17T01:58:09.525 INFO:teuthology.orchestra.run.smithi013.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:58:09.526 INFO:teuthology.orchestra.run.smithi013.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:58:09.527 INFO:teuthology.orchestra.run.smithi013.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-17T01:58:09.528 INFO:teuthology.orchestra.run.smithi013.stdout:anonymous 2021-09-17T01:58:09.528 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-17T01:58:09.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi013.front.sepia.ceph.com:8000', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--url', 'http://smithi013.front.sepia.ceph.com:8000', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:58:09.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi013.front.sepia.ceph.com:8000', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--url', 'http://smithi013.front.sepia.ceph.com:8000', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:58:09.529 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi013.front.sepia.ceph.com:8000 --access-key HXCQHMJWFX35AU7N --secret B32eJkOEzpLbQQLCICRnQRj2OnvubCoS --url http://smithi013.front.sepia.ceph.com:8000 --access-key HXCQHMJWFX35AU7N --secret B32eJkOEzpLbQQLCICRnQRj2OnvubCoS --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T01:58:09.588 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:09.588 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:09.621 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T01:58:09.619+0000 7f62a59e4700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.701b5af2-9aae-4e89-bd12-7ed067bbad31.control err=-107 2021-09-17T01:58:09.630 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:58:09.630 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:58:09.630 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-realm", 2021-09-17T01:58:09.631 INFO:teuthology.orchestra.run.smithi045.stdout: "current_period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:09.631 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 2 2021-09-17T01:58:09.631 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:58:09.632 INFO:tasks.rgw_multisite:creating zone.. 2021-09-17T01:58:09.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--endpoints', 'http://smithi045.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:58:09.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HXCQHMJWFX35AU7N', '--secret', 'B32eJkOEzpLbQQLCICRnQRj2OnvubCoS', '--endpoints', 'http://smithi045.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:58:09.634 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HXCQHMJWFX35AU7N --secret B32eJkOEzpLbQQLCICRnQRj2OnvubCoS --endpoints http://smithi045.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T01:58:09.689 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:09.690 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:09.713 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:58:09.712+0000 7f474f818380 0 failed reading obj info from .rgw.root:zone_info.505278ea-3c6d-4c68-8442-578099047a49: (2) No such file or directory 2021-09-17T01:58:09.714 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:58:09.712+0000 7f474f818380 0 WARNING: could not read zone params for zone id=505278ea-3c6d-4c68-8442-578099047a49 name=test-zone3 2021-09-17T01:58:09.714 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:58:09.712+0000 7f474f818380 0 failed reading obj info from .rgw.root:zone_info.95b0d488-6c59-4a37-b3ed-d5fa65f27678: (2) No such file or directory 2021-09-17T01:58:09.714 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:58:09.712+0000 7f474f818380 0 WARNING: could not read zone params for zone id=95b0d488-6c59-4a37-b3ed-d5fa65f27678 name=test-zone1 2021-09-17T01:58:09.717 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:58:09.718 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "e8223d60-810a-44ad-a8ee-67d9caa7bcb0", 2021-09-17T01:58:09.718 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone4", 2021-09-17T01:58:09.718 INFO:teuthology.orchestra.run.smithi045.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-17T01:58:09.718 INFO:teuthology.orchestra.run.smithi045.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-17T01:58:09.719 INFO:teuthology.orchestra.run.smithi045.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-17T01:58:09.719 INFO:teuthology.orchestra.run.smithi045.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-17T01:58:09.719 INFO:teuthology.orchestra.run.smithi045.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-17T01:58:09.719 INFO:teuthology.orchestra.run.smithi045.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-17T01:58:09.719 INFO:teuthology.orchestra.run.smithi045.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-17T01:58:09.720 INFO:teuthology.orchestra.run.smithi045.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-17T01:58:09.720 INFO:teuthology.orchestra.run.smithi045.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-17T01:58:09.720 INFO:teuthology.orchestra.run.smithi045.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-17T01:58:09.720 INFO:teuthology.orchestra.run.smithi045.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-17T01:58:09.720 INFO:teuthology.orchestra.run.smithi045.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-17T01:58:09.721 INFO:teuthology.orchestra.run.smithi045.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-17T01:58:09.721 INFO:teuthology.orchestra.run.smithi045.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-17T01:58:09.721 INFO:teuthology.orchestra.run.smithi045.stdout: "system_key": { 2021-09-17T01:58:09.721 INFO:teuthology.orchestra.run.smithi045.stdout: "access_key": "HXCQHMJWFX35AU7N", 2021-09-17T01:58:09.721 INFO:teuthology.orchestra.run.smithi045.stdout: "secret_key": "B32eJkOEzpLbQQLCICRnQRj2OnvubCoS" 2021-09-17T01:58:09.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:09.722 INFO:teuthology.orchestra.run.smithi045.stdout: "placement_pools": [ 2021-09-17T01:58:09.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:09.722 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "default-placement", 2021-09-17T01:58:09.723 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:09.723 INFO:teuthology.orchestra.run.smithi045.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-17T01:58:09.723 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_classes": { 2021-09-17T01:58:09.723 INFO:teuthology.orchestra.run.smithi045.stdout: "STANDARD": { 2021-09-17T01:58:09.723 INFO:teuthology.orchestra.run.smithi045.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2021-09-17T01:58:09.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:09.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:09.724 INFO:teuthology.orchestra.run.smithi045.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-17T01:58:09.724 INFO:teuthology.orchestra.run.smithi045.stdout: "index_type": 0 2021-09-17T01:58:09.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:09.725 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:09.725 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:58:09.725 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_config": { 2021-09-17T01:58:09.725 INFO:teuthology.orchestra.run.smithi045.stdout: "event_retention_days": 7, 2021-09-17T01:58:09.725 INFO:teuthology.orchestra.run.smithi045.stdout: "start_with_full_sync": false 2021-09-17T01:58:09.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:09.726 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:58:09.726 INFO:teuthology.orchestra.run.smithi045.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-17T01:58:09.726 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:58:09.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-09-17T01:58:10.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-09-17T01:58:10.562 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-09-17T01:58:10.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-09-17T01:58:11.574 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-09-17T01:58:11.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:58:11.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:58:11.591 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T01:58:11.639 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:11.639 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:11.664 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:58:11.664 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "e8223d60-810a-44ad-a8ee-67d9caa7bcb0", 2021-09-17T01:58:11.664 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone4", 2021-09-17T01:58:11.664 INFO:teuthology.orchestra.run.smithi045.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-17T01:58:11.665 INFO:teuthology.orchestra.run.smithi045.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-17T01:58:11.665 INFO:teuthology.orchestra.run.smithi045.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-17T01:58:11.665 INFO:teuthology.orchestra.run.smithi045.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-17T01:58:11.666 INFO:teuthology.orchestra.run.smithi045.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-17T01:58:11.666 INFO:teuthology.orchestra.run.smithi045.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-17T01:58:11.666 INFO:teuthology.orchestra.run.smithi045.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-17T01:58:11.666 INFO:teuthology.orchestra.run.smithi045.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-17T01:58:11.667 INFO:teuthology.orchestra.run.smithi045.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-17T01:58:11.667 INFO:teuthology.orchestra.run.smithi045.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-17T01:58:11.667 INFO:teuthology.orchestra.run.smithi045.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-17T01:58:11.667 INFO:teuthology.orchestra.run.smithi045.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-17T01:58:11.668 INFO:teuthology.orchestra.run.smithi045.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-17T01:58:11.668 INFO:teuthology.orchestra.run.smithi045.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-17T01:58:11.668 INFO:teuthology.orchestra.run.smithi045.stdout: "system_key": { 2021-09-17T01:58:11.668 INFO:teuthology.orchestra.run.smithi045.stdout: "access_key": "HXCQHMJWFX35AU7N", 2021-09-17T01:58:11.668 INFO:teuthology.orchestra.run.smithi045.stdout: "secret_key": "B32eJkOEzpLbQQLCICRnQRj2OnvubCoS" 2021-09-17T01:58:11.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:11.669 INFO:teuthology.orchestra.run.smithi045.stdout: "placement_pools": [ 2021-09-17T01:58:11.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:11.670 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "default-placement", 2021-09-17T01:58:11.670 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:11.670 INFO:teuthology.orchestra.run.smithi045.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-17T01:58:11.670 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_classes": { 2021-09-17T01:58:11.671 INFO:teuthology.orchestra.run.smithi045.stdout: "STANDARD": { 2021-09-17T01:58:11.671 INFO:teuthology.orchestra.run.smithi045.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2021-09-17T01:58:11.671 INFO:teuthology.orchestra.run.smithi045.stdout: "compression_type": "random" 2021-09-17T01:58:11.671 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:11.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:11.672 INFO:teuthology.orchestra.run.smithi045.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-17T01:58:11.672 INFO:teuthology.orchestra.run.smithi045.stdout: "index_type": 0 2021-09-17T01:58:11.672 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:11.672 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:11.673 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:58:11.673 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_config": { 2021-09-17T01:58:11.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event_retention_days": 7, 2021-09-17T01:58:11.673 INFO:teuthology.orchestra.run.smithi045.stdout: "start_with_full_sync": false 2021-09-17T01:58:11.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:11.674 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "", 2021-09-17T01:58:11.674 INFO:teuthology.orchestra.run.smithi045.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-17T01:58:11.674 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:58:11.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:58:11.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:58:11.676 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T01:58:11.718 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:11.719 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:11.727 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T01:58:11.726+0000 7f0372bbf380 0 period (3c964c05-2a9c-48af-9858-975bf7b57c1a does not have zone e8223d60-810a-44ad-a8ee-67d9caa7bcb0 configured 2021-09-17T01:58:19.621 INFO:teuthology.orchestra.run.smithi045.stderr:Sending period to new master zone 95b0d488-6c59-4a37-b3ed-d5fa65f27678 2021-09-17T01:58:19.648 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:58:19.648 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:19.649 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 4, 2021-09-17T01:58:19.649 INFO:teuthology.orchestra.run.smithi045.stdout: "predecessor_uuid": "d9cd66ce-d16c-4226-9f7c-60489089c5d1", 2021-09-17T01:58:19.649 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": [], 2021-09-17T01:58:19.649 INFO:teuthology.orchestra.run.smithi045.stdout: "period_map": { 2021-09-17T01:58:19.650 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:19.650 INFO:teuthology.orchestra.run.smithi045.stdout: "zonegroups": [ 2021-09-17T01:58:19.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:19.650 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "test-zonegroup", 2021-09-17T01:58:19.651 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zonegroup", 2021-09-17T01:58:19.651 INFO:teuthology.orchestra.run.smithi045.stdout: "api_name": "", 2021-09-17T01:58:19.651 INFO:teuthology.orchestra.run.smithi045.stdout: "is_master": "true", 2021-09-17T01:58:19.651 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T01:58:19.651 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T01:58:19.652 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:58:19.652 INFO:teuthology.orchestra.run.smithi045.stdout: "hostnames": [], 2021-09-17T01:58:19.652 INFO:teuthology.orchestra.run.smithi045.stdout: "hostnames_s3website": [], 2021-09-17T01:58:19.652 INFO:teuthology.orchestra.run.smithi045.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:58:19.652 INFO:teuthology.orchestra.run.smithi045.stdout: "zones": [ 2021-09-17T01:58:19.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:19.653 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T01:58:19.653 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone3", 2021-09-17T01:58:19.653 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T01:58:19.654 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi013.front.sepia.ceph.com:8001" 2021-09-17T01:58:19.654 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:58:19.654 INFO:teuthology.orchestra.run.smithi045.stdout: "log_meta": "false", 2021-09-17T01:58:19.654 INFO:teuthology.orchestra.run.smithi045.stdout: "log_data": "true", 2021-09-17T01:58:19.654 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_index_max_shards": 11, 2021-09-17T01:58:19.655 INFO:teuthology.orchestra.run.smithi045.stdout: "read_only": "false", 2021-09-17T01:58:19.655 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_type": "", 2021-09-17T01:58:19.655 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from_all": "true", 2021-09-17T01:58:19.655 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from": [], 2021-09-17T01:58:19.656 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_zone": "" 2021-09-17T01:58:19.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:19.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:19.656 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T01:58:19.656 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone2", 2021-09-17T01:58:19.657 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T01:58:19.657 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi045.front.sepia.ceph.com:8000" 2021-09-17T01:58:19.657 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:58:19.657 INFO:teuthology.orchestra.run.smithi045.stdout: "log_meta": "false", 2021-09-17T01:58:19.658 INFO:teuthology.orchestra.run.smithi045.stdout: "log_data": "true", 2021-09-17T01:58:19.658 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_index_max_shards": 11, 2021-09-17T01:58:19.658 INFO:teuthology.orchestra.run.smithi045.stdout: "read_only": "false", 2021-09-17T01:58:19.658 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_type": "", 2021-09-17T01:58:19.659 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from_all": "true", 2021-09-17T01:58:19.659 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from": [], 2021-09-17T01:58:19.659 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_zone": "" 2021-09-17T01:58:19.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:19.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:19.660 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:58:19.660 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone1", 2021-09-17T01:58:19.660 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T01:58:19.660 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T01:58:19.661 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:58:19.661 INFO:teuthology.orchestra.run.smithi045.stdout: "log_meta": "false", 2021-09-17T01:58:19.661 INFO:teuthology.orchestra.run.smithi045.stdout: "log_data": "true", 2021-09-17T01:58:19.661 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_index_max_shards": 11, 2021-09-17T01:58:19.661 INFO:teuthology.orchestra.run.smithi045.stdout: "read_only": "false", 2021-09-17T01:58:19.662 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_type": "", 2021-09-17T01:58:19.662 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from_all": "true", 2021-09-17T01:58:19.662 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from": [], 2021-09-17T01:58:19.662 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_zone": "" 2021-09-17T01:58:19.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:19.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:19.663 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "e8223d60-810a-44ad-a8ee-67d9caa7bcb0", 2021-09-17T01:58:19.663 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone4", 2021-09-17T01:58:19.663 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T01:58:19.664 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi045.front.sepia.ceph.com:8001" 2021-09-17T01:58:19.664 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:58:19.664 INFO:teuthology.orchestra.run.smithi045.stdout: "log_meta": "false", 2021-09-17T01:58:19.664 INFO:teuthology.orchestra.run.smithi045.stdout: "log_data": "true", 2021-09-17T01:58:19.665 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_index_max_shards": 11, 2021-09-17T01:58:19.665 INFO:teuthology.orchestra.run.smithi045.stdout: "read_only": "false", 2021-09-17T01:58:19.665 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_type": "pubsub", 2021-09-17T01:58:19.665 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from_all": "false", 2021-09-17T01:58:19.666 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from": [ 2021-09-17T01:58:19.666 INFO:teuthology.orchestra.run.smithi045.stdout: "test-zone1" 2021-09-17T01:58:19.666 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:58:19.666 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_zone": "" 2021-09-17T01:58:19.666 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:19.667 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:58:19.667 INFO:teuthology.orchestra.run.smithi045.stdout: "placement_targets": [ 2021-09-17T01:58:19.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:19.667 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "default-placement", 2021-09-17T01:58:19.667 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2021-09-17T01:58:19.668 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_classes": [ 2021-09-17T01:58:19.668 INFO:teuthology.orchestra.run.smithi045.stdout: "STANDARD" 2021-09-17T01:58:19.668 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:58:19.668 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:19.669 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:58:19.669 INFO:teuthology.orchestra.run.smithi045.stdout: "default_placement": "default-placement", 2021-09-17T01:58:19.669 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:58:19.669 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_policy": { 2021-09-17T01:58:19.669 INFO:teuthology.orchestra.run.smithi045.stdout: "groups": [] 2021-09-17T01:58:19.670 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:19.670 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:19.670 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T01:58:19.670 INFO:teuthology.orchestra.run.smithi045.stdout: "short_zone_ids": [ 2021-09-17T01:58:19.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:19.672 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T01:58:19.672 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 4005434042 2021-09-17T01:58:19.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:19.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:19.673 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T01:58:19.673 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 4091795418 2021-09-17T01:58:19.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:19.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:19.674 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:58:19.674 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 803492335 2021-09-17T01:58:19.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:19.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:19.675 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "e8223d60-810a-44ad-a8ee-67d9caa7bcb0", 2021-09-17T01:58:19.675 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 3317638424 2021-09-17T01:58:19.675 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:19.675 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:58:19.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:19.676 INFO:teuthology.orchestra.run.smithi045.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T01:58:19.676 INFO:teuthology.orchestra.run.smithi045.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T01:58:19.676 INFO:teuthology.orchestra.run.smithi045.stdout: "period_config": { 2021-09-17T01:58:19.677 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_quota": { 2021-09-17T01:58:19.677 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": false, 2021-09-17T01:58:19.677 INFO:teuthology.orchestra.run.smithi045.stdout: "check_on_raw": false, 2021-09-17T01:58:19.677 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size": -1, 2021-09-17T01:58:19.678 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size_kb": 0, 2021-09-17T01:58:19.678 INFO:teuthology.orchestra.run.smithi045.stdout: "max_objects": -1 2021-09-17T01:58:19.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:19.678 INFO:teuthology.orchestra.run.smithi045.stdout: "user_quota": { 2021-09-17T01:58:19.678 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": false, 2021-09-17T01:58:19.679 INFO:teuthology.orchestra.run.smithi045.stdout: "check_on_raw": false, 2021-09-17T01:58:19.679 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size": -1, 2021-09-17T01:58:19.679 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size_kb": 0, 2021-09-17T01:58:19.679 INFO:teuthology.orchestra.run.smithi045.stdout: "max_objects": -1 2021-09-17T01:58:19.680 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:19.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:19.680 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T01:58:19.680 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_name": "test-realm", 2021-09-17T01:58:19.680 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:19.681 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:58:19.681 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-09-17T01:58:19.682 INFO:teuthology.orchestra.run:waiting for 300 2021-09-17T01:58:19.714 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T01:58:19.712+0000 7f13e7ff4700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 36486) UID: 0 2021-09-17T01:58:19.714 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T01:58:19.712+0000 7f13f8e95440 -1 shutting down 2021-09-17T01:58:20.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-09-17T01:58:25.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~6s 2021-09-17T01:58:25.783 INFO:tasks.rgw.c2.client.1:Stopped 2021-09-17T01:58:25.784 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-17T01:58:25.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-17T01:58:25.788 INFO:tasks.rgw.c2.client.1:Started 2021-09-17T01:58:25.788 DEBUG:teuthology.orchestra.run.smithi045:> curl http://smithi045.front.sepia.ceph.com:8001 2021-09-17T01:58:25.810 INFO:teuthology.orchestra.run.smithi045.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:58:25.811 INFO:teuthology.orchestra.run.smithi045.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:58:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-17T01:58:25.812 INFO:teuthology.orchestra.run.smithi045.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi045.front.sepia.ceph.com port 8001: Connection refused 2021-09-17T01:58:26.814 DEBUG:teuthology.orchestra.run.smithi045:> curl http://smithi045.front.sepia.ceph.com:8001 2021-09-17T01:58:26.834 INFO:teuthology.orchestra.run.smithi045.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T01:58:26.834 INFO:teuthology.orchestra.run.smithi045.stderr: Dload Upload Total Spent Left Speed 2021-09-17T01:58:26.836 INFO:teuthology.orchestra.run.smithi045.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 130k 0 --:--:-- --:--:-- --:--:-- 130k 2021-09-17T01:58:26.836 INFO:teuthology.orchestra.run.smithi045.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-006143f642-10e9-test-zone4","HostId":"10e9-test-zone4-test-zonegroup"} 2021-09-17T01:58:26.837 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-09-17T01:58:26.837 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-09-17T01:58:26.850 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph-c_2a911e49300877ce783c2becf9f484a85fa1c049/qa/../src/test/rgw/rgw_multi 2021-09-17T01:58:26.864 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-09-17T01:58:26.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:58:26.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T01:58:26.865 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T01:58:26.919 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:26.919 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:26.968 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T01:58:26.969 INFO:teuthology.orchestra.run.smithi013.stdout: "user_id": "rgw-multisite-test-user", 2021-09-17T01:58:26.969 INFO:teuthology.orchestra.run.smithi013.stdout: "display_name": "Multisite Test User", 2021-09-17T01:58:26.969 INFO:teuthology.orchestra.run.smithi013.stdout: "email": "", 2021-09-17T01:58:26.969 INFO:teuthology.orchestra.run.smithi013.stdout: "suspended": 0, 2021-09-17T01:58:26.969 INFO:teuthology.orchestra.run.smithi013.stdout: "max_buckets": 1000, 2021-09-17T01:58:26.970 INFO:teuthology.orchestra.run.smithi013.stdout: "subusers": [], 2021-09-17T01:58:26.970 INFO:teuthology.orchestra.run.smithi013.stdout: "keys": [ 2021-09-17T01:58:26.970 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:26.970 INFO:teuthology.orchestra.run.smithi013.stdout: "user": "rgw-multisite-test-user", 2021-09-17T01:58:26.971 INFO:teuthology.orchestra.run.smithi013.stdout: "access_key": "H3P37FIH5X5I87STHGBO", 2021-09-17T01:58:26.971 INFO:teuthology.orchestra.run.smithi013.stdout: "secret_key": "vnTDPIAiSoVc4yK9zS7woTAlNt2RM4EOxcvhJfPS" 2021-09-17T01:58:26.971 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:58:26.971 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T01:58:26.972 INFO:teuthology.orchestra.run.smithi013.stdout: "swift_keys": [], 2021-09-17T01:58:26.972 INFO:teuthology.orchestra.run.smithi013.stdout: "caps": [], 2021-09-17T01:58:26.972 INFO:teuthology.orchestra.run.smithi013.stdout: "op_mask": "read, write, delete", 2021-09-17T01:58:26.972 INFO:teuthology.orchestra.run.smithi013.stdout: "default_placement": "", 2021-09-17T01:58:26.972 INFO:teuthology.orchestra.run.smithi013.stdout: "default_storage_class": "", 2021-09-17T01:58:26.973 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_tags": [], 2021-09-17T01:58:26.973 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_quota": { 2021-09-17T01:58:26.973 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T01:58:26.973 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T01:58:26.973 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T01:58:26.974 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T01:58:26.974 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T01:58:26.974 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:26.974 INFO:teuthology.orchestra.run.smithi013.stdout: "user_quota": { 2021-09-17T01:58:26.975 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T01:58:26.975 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T01:58:26.975 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T01:58:26.975 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T01:58:26.975 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T01:58:26.976 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:26.976 INFO:teuthology.orchestra.run.smithi013.stdout: "temp_url_keys": [], 2021-09-17T01:58:26.976 INFO:teuthology.orchestra.run.smithi013.stdout: "type": "rgw", 2021-09-17T01:58:26.976 INFO:teuthology.orchestra.run.smithi013.stdout: "mfa_ids": [] 2021-09-17T01:58:26.976 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T01:58:26.977 INFO:teuthology.orchestra.run.smithi013.stdout: 2021-09-17T01:58:26.982 INFO:rgw_multi.tests:meta checkpoint 2021-09-17T01:58:26.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:26.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:26.983 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:58:27.026 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:27.026 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:27.063 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T01:58:27.075 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T01:58:27.075 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:27.075 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631843906.965677_70.1", 2021-09-17T01:58:27.075 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T01:58:26.965677Z" 2021-09-17T01:58:27.076 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:27.076 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:27.076 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T01:58:27.076 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T01:58:27.076 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:27.077 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:27.077 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T01:58:27.077 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T01:58:27.077 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T01:58:27.077 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T01:58:27.078 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T01:58:27.078 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T01:58:27.078 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T01:58:27.078 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T01:58:27.079 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T01:58:27.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:27.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:27.080 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:58:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:27.180 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:58:27.180 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:58:27.180 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:58:27.181 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:58:27.181 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:58:27.181 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:27.181 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:27.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:27.182 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:58:27.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:27.182 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:58:27.182 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:27.183 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:27.183 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:27.183 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:27.183 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:58:27.184 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:27.184 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:27.184 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:27.184 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:27.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:27.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:27.185 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:58:27.185 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:27.185 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:27.186 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:27.186 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:27.186 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:27.186 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:27.187 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:27.187 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:27.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:27.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:27.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:27.188 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:58:27.188 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:27.188 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:27.188 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:27.188 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:27.189 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:27.189 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:27.189 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:27.189 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:27.190 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:27.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:27.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:27.190 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:58:27.190 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:27.192 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:27.192 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:27.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:27.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:27.193 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:58:27.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:27.193 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:58:27.193 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:58:27.193 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:58:27.194 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:27.194 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:58:27.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:58:31.487 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-09-17T01:58:32.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:32.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:32.197 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:58:32.246 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:32.247 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:32.299 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:58:32.299 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:58:32.299 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:58:32.300 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:58:32.300 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:58:32.300 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:32.300 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:32.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:32.301 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:58:32.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:32.301 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:58:32.301 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:32.301 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:32.302 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:32.302 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:32.302 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:58:32.302 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:32.303 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:32.303 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:32.303 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:32.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:32.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:32.303 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:58:32.304 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:32.304 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:32.304 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:32.304 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:32.305 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:32.305 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:32.305 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:32.305 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:32.305 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:32.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:32.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:32.306 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:58:32.306 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:32.306 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:32.307 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:32.307 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:32.307 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:32.307 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:32.307 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:32.308 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:32.308 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:32.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:32.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:32.308 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:58:32.309 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:32.309 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:32.309 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:32.309 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:32.309 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:32.309 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:32.310 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:32.310 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:32.310 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:32.310 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:32.311 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:58:32.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:32.311 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:58:32.311 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:58:32.311 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:58:32.312 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:32.312 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:58:32.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:58:37.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:37.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:37.316 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:58:37.360 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:37.360 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:37.409 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:58:37.410 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:58:37.410 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:58:37.410 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:58:37.411 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:58:37.411 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:37.411 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:37.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:37.411 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:58:37.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:37.412 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:58:37.412 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:37.412 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:37.413 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:37.413 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:37.413 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:58:37.413 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:37.414 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:37.414 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:37.414 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:37.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:37.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:37.415 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:58:37.415 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:37.415 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:37.415 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:37.416 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:37.416 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:37.416 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:37.416 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:37.416 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:37.417 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:37.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:37.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:37.417 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:58:37.417 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:37.418 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:37.418 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:37.418 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:37.418 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:37.419 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:37.419 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:37.419 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:37.419 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:37.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:37.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:37.420 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:58:37.420 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:37.420 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:37.421 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:37.421 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:37.421 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:37.421 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:37.421 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:37.422 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:37.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:37.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:37.422 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:58:37.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:37.423 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:58:37.423 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:58:37.423 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:58:37.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:37.424 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:58:37.425 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:58:42.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:42.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:42.428 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:58:42.477 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:42.477 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:42.530 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:58:42.531 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:58:42.531 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:58:42.531 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:58:42.531 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:58:42.532 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:42.532 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:42.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:42.532 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:58:42.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:42.533 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:58:42.533 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:42.533 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:42.533 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:42.534 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:42.534 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:58:42.534 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:42.534 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:42.535 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:42.535 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:42.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:42.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:42.535 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:58:42.536 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:42.536 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:42.536 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:42.536 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:42.536 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:42.537 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:42.537 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:42.537 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:42.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:42.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:42.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:42.538 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:58:42.538 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:42.538 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:42.539 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:42.539 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:42.539 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:42.539 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:42.539 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:42.540 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:42.540 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:42.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:42.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:42.541 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:58:42.541 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:42.541 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:42.541 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:42.541 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:42.541 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:42.542 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:42.542 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:42.542 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:42.542 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:42.543 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:42.543 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:58:42.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:42.543 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:58:42.544 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:58:42.544 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:58:42.544 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:42.544 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:58:42.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:58:47.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:47.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:47.549 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:58:47.600 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:47.600 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:47.653 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:58:47.653 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:58:47.654 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:58:47.654 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:58:47.654 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:58:47.654 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:47.655 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:47.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:47.655 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:58:47.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:47.655 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:58:47.656 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:47.656 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:47.656 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:47.656 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:47.656 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:58:47.657 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:47.657 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:47.657 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:47.657 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:47.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:47.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:47.658 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:58:47.658 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:47.658 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:47.658 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:47.659 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:47.659 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:47.659 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:47.659 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:47.659 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:47.660 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:47.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:47.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:47.660 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:58:47.660 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:47.661 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:47.661 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:47.661 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:47.661 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:47.661 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:47.662 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:47.662 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:47.662 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:47.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:47.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:47.663 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:58:47.663 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:47.663 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:47.663 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:47.663 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:47.664 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:47.664 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:47.664 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:47.664 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:47.664 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:47.665 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:47.665 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:58:47.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:47.665 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:58:47.665 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:58:47.666 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:58:47.666 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:47.666 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:58:47.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:58:52.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:52.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:52.670 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:58:52.721 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:52.722 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:52.774 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:58:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:58:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:58:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:58:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:58:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:58:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:58:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:58:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:58:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:58:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:58:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:58:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:58:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:58:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:58:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:52.788 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:58:52.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:58:57.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:57.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:58:57.792 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:58:57.843 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:58:57.844 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:58:57.896 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:58:57.897 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:58:57.897 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:58:57.897 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:58:57.898 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:58:57.898 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:58:57.898 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:57.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:57.898 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:58:57.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:57.899 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:58:57.899 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:57.899 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:57.900 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:57.900 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:57.900 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:58:57.900 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:57.900 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:57.901 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:57.901 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:57.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:57.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:57.901 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:58:57.902 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:57.902 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:57.902 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:57.902 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:57.903 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:57.903 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:57.903 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:57.903 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:57.903 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:57.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:57.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:57.904 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:58:57.904 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:57.904 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:57.905 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:57.905 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:57.905 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:57.905 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:57.905 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:57.906 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:57.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:57.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:57.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:58:57.907 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:58:57.907 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:58:57.907 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:58:57.907 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:58:57.907 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:58:57.908 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:58:57.908 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:58:57.908 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:58:57.908 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:58:57.908 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:57.909 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:57.909 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:58:57.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:58:57.909 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:58:57.910 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:58:57.910 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:58:57.910 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:58:57.910 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:58:57.911 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:02.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:02.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:02.913 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:02.964 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:02.964 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:03.017 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:03.017 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:03.017 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:03.018 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:03.018 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:03.018 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:03.019 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:03.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:03.019 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:03.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:03.019 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:03.020 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:03.020 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:03.020 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:03.020 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:03.020 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:03.021 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:03.021 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:03.021 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:03.021 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:03.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:03.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:03.022 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:03.022 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:03.022 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:03.023 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:03.023 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:03.023 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:03.023 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:03.023 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:03.024 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:03.024 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:03.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:03.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:03.024 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:03.025 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:03.025 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:03.025 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:03.025 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:03.026 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:03.026 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:03.026 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:03.026 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:03.026 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:03.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:03.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:03.027 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:03.027 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:03.027 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:03.028 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:03.028 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:03.028 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:03.028 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:03.028 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:03.029 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:03.029 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:03.029 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:03.029 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:03.029 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:03.030 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:03.030 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:03.030 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:03.030 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:03.031 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:03.032 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:08.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:08.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:08.034 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:08.083 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:08.083 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:08.137 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:08.137 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:08.138 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:08.139 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:08.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:08.139 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:08.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:08.139 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:08.140 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:08.140 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:08.140 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:08.140 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:08.141 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:08.141 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:08.141 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:08.141 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:08.141 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:08.142 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:08.142 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:08.143 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:08.143 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:08.143 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:08.143 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:08.144 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:08.144 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:08.144 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:08.144 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:08.144 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:08.145 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:08.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:08.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:08.145 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:08.146 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:08.146 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:08.146 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:08.146 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:08.146 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:08.147 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:08.147 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:08.147 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:08.147 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:08.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:08.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:08.148 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:08.148 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:08.149 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:08.149 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:08.149 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:08.149 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:08.149 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:08.150 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:08.150 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:08.150 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:08.150 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:08.150 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:08.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:08.151 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:08.151 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:08.151 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:08.151 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:08.152 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:08.153 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:13.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:13.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:13.155 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:13.205 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:13.205 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:13.259 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:13.259 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:13.260 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:13.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:13.262 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:13.263 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:13.264 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:13.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:13.266 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:13.267 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:13.268 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:13.269 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:13.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:13.271 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:13.272 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:13.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:13.274 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:13.275 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:13.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:13.277 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:13.277 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:13.278 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:18.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:18.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:18.280 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:18.331 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:18.384 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:18.385 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:18.385 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:18.385 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:18.385 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:18.386 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:18.386 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:18.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:18.386 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:18.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:18.387 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:18.387 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:18.387 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:18.387 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:18.388 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:18.388 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:18.388 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:18.388 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:18.389 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:18.389 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:18.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:18.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:18.389 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:18.390 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:18.390 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:18.390 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:18.390 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:18.390 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:18.391 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:18.391 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:18.391 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:18.391 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:18.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:18.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:18.392 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:18.392 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:18.392 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:18.393 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:18.393 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:18.393 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:18.393 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:18.393 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:18.394 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:18.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:18.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:18.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:18.394 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:18.395 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:18.395 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:18.395 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:18.395 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:18.395 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:18.396 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:18.396 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:18.396 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:18.396 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:18.397 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:18.397 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:18.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:18.397 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:18.397 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:18.398 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:18.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:18.398 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:18.399 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:23.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:23.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:23.401 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:23.450 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:23.503 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:23.503 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:23.503 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:23.503 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:23.504 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:23.504 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:23.504 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:23.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:23.505 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:23.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:23.505 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:23.506 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:23.506 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:23.506 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:23.506 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:23.506 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:23.507 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:23.507 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:23.507 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:23.507 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:23.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:23.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:23.508 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:23.508 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:23.509 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:23.509 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:23.509 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:23.509 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:23.510 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:23.510 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:23.510 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:23.510 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:23.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:23.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:23.511 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:23.511 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:23.511 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:23.512 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:23.512 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:23.512 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:23.512 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:23.512 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:23.513 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:23.513 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:23.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:23.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:23.514 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:23.514 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:23.514 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:23.514 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:23.514 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:23.515 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:23.515 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:23.515 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:23.515 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:23.516 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:23.516 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:23.516 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:23.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:23.516 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:23.517 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:23.517 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:23.517 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:23.517 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:23.518 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:28.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:28.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:28.521 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:28.626 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:28.627 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:28.627 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:28.627 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:28.628 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:28.628 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:28.628 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:28.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:28.629 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:28.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:28.629 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:28.629 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:28.630 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:28.630 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:28.630 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:28.630 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:28.631 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:28.631 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:28.631 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:28.631 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:28.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:28.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:28.632 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:28.632 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:28.633 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:28.633 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:28.633 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:28.633 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:28.633 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:28.634 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:28.634 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:28.634 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:28.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:28.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:28.635 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:28.635 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:28.635 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:28.636 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:28.636 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:28.636 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:28.636 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:28.636 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:28.637 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:28.637 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:28.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:28.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:28.638 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:28.638 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:28.638 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:28.638 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:28.639 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:28.639 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:28.639 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:28.639 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:28.640 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:28.640 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:28.640 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:28.640 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:28.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:28.641 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:28.641 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:28.641 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:28.641 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:28.642 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:28.643 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:33.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:33.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:33.645 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:33.695 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:33.696 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:33.748 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:33.749 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:33.750 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:33.751 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:33.752 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:33.753 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:33.754 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:33.754 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:33.754 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:33.754 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:33.755 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:33.755 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:33.755 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:33.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:33.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:33.756 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:33.756 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:33.756 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:33.756 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:33.756 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:33.757 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:33.757 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:33.757 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:33.757 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:33.758 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:33.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:33.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:33.758 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:33.759 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:33.759 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:33.759 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:33.759 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:33.760 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:33.760 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:33.760 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:33.760 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:33.760 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:33.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:33.761 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:33.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:33.761 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:33.761 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:33.762 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:33.762 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:33.762 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:33.763 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:38.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:38.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:38.766 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:38.817 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:38.817 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:38.870 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:38.871 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:38.871 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:38.871 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:38.871 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:38.871 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:38.872 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:38.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:38.872 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:38.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:38.873 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:38.873 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:38.873 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:38.873 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:38.874 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:38.874 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:38.874 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:38.874 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:38.875 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:38.875 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:38.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:38.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:38.875 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:38.876 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:38.876 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:38.876 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:38.876 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:38.877 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:38.877 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:38.877 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:38.877 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:38.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:38.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:38.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:38.878 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:38.878 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:38.879 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:38.879 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:38.879 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:38.879 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:38.879 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:38.880 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:38.880 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:38.880 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:38.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:38.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:38.881 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:38.881 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:38.881 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:38.881 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:38.882 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:38.882 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:38.882 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:38.882 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:38.882 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:38.883 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:38.883 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:38.883 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:38.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:38.884 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:38.884 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:38.884 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:38.884 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:38.884 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:38.886 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:43.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:43.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:43.888 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:43.939 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:43.939 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:43.993 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:43.993 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:43.994 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:43.995 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:43.996 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:43.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:43.998 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:43.999 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:44.000 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:44.000 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:44.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:44.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:44.001 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:44.002 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:44.003 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:44.004 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:44.005 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:44.006 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:44.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:44.007 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:44.008 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:49.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:49.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:49.010 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:49.061 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:49.061 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:49.113 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:49.114 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:49.114 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:49.114 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:49.114 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:49.115 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:49.115 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:49.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:49.115 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:49.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:49.116 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:49.116 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:49.116 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:49.116 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:49.117 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:49.117 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:49.117 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:49.117 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:49.117 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:49.118 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:49.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:49.118 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:49.118 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:49.119 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:49.119 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:49.119 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:49.119 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:49.119 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:49.120 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:49.120 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:49.120 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:49.120 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:49.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:49.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:49.121 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:49.121 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:49.121 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:49.121 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:49.122 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:49.122 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:49.122 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:49.122 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:49.123 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:49.123 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:49.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:49.123 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:49.123 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:49.124 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:49.124 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:49.124 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:49.124 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:49.124 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:49.125 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:49.125 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:49.125 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:49.125 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:49.126 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:49.126 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:49.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:49.126 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:49.127 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:49.127 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:49.127 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:49.127 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:49.128 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:54.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:54.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:54.131 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:54.182 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:54.183 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:54.237 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:54.237 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:54.237 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:54.237 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:54.238 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:54.238 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:54.238 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:54.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:54.239 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:54.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:54.239 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:54.239 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:54.240 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:54.240 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:54.240 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:54.240 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:54.241 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:54.241 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:54.241 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:54.241 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:54.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:54.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:54.242 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:54.242 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:54.243 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:54.243 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:54.243 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:54.243 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:54.243 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:54.244 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:54.244 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:54.244 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:54.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:54.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:54.245 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:54.245 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:54.246 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:54.246 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:54.246 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:54.246 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:54.247 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:54.247 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:54.247 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:54.247 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:54.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:54.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:54.248 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:54.248 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:54.249 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:54.249 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:54.249 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:54.249 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:54.249 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:54.250 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:54.250 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:54.250 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:54.251 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:54.251 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:54.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:54.251 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:54.251 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:54.252 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:54.252 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:54.252 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:54.254 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T01:59:59.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:59.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:59:59.256 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:59:59.306 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:59:59.307 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:59:59.359 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T01:59:59.360 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T01:59:59.360 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T01:59:59.360 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T01:59:59.360 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T01:59:59.361 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T01:59:59.361 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:59.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:59.361 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T01:59:59.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:59.362 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T01:59:59.362 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:59.362 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:59.362 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:59.363 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:59.363 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T01:59:59.363 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:59.363 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:59.364 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:59.364 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:59.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:59.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:59.364 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T01:59:59.365 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:59.365 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:59.365 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:59.365 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:59.365 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:59.366 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:59.366 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:59.366 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:59.366 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:59.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:59.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:59.367 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T01:59:59.367 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:59.367 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:59.368 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:59.368 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:59.368 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:59.368 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:59.369 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:59.369 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:59.369 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:59.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:59.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T01:59:59.370 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T01:59:59.370 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T01:59:59.370 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T01:59:59.370 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T01:59:59.370 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T01:59:59.371 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T01:59:59.371 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T01:59:59.371 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T01:59:59.371 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T01:59:59.372 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:59.372 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:59.372 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T01:59:59.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T01:59:59.372 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T01:59:59.373 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T01:59:59.373 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T01:59:59.373 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T01:59:59.373 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T01:59:59.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T02:00:04.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:04.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:04.377 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:04.429 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:04.429 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:04.483 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:04.483 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:04.484 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:04.484 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:04.484 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:04.484 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:04.485 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:04.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:04.485 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:04.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:04.486 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:04.486 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:04.486 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:04.486 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:04.487 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:04.487 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:04.487 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:04.487 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:04.487 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:04.488 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:04.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:04.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:04.488 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:04.489 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:04.489 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:04.489 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:04.489 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:04.490 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:04.490 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:04.490 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:04.490 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:04.490 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:04.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:04.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:04.491 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:04.491 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:04.492 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:04.492 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:04.492 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:04.492 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:04.492 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:04.493 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:04.493 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:04.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:04.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:04.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:04.494 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:04.495 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:04.495 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:04.495 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:04.495 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:04.495 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:04.496 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:04.496 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:04.496 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:04.496 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:04.496 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:04.497 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:04.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:04.497 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:04.497 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:04.498 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:04.498 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:04.498 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:04.500 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T02:00:09.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:09.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:09.502 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:09.553 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:09.553 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:09.609 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:09.610 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:09.610 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:09.610 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:09.610 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:09.611 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:09.611 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:09.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:09.611 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:09.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:09.612 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:09.612 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:09.612 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:09.612 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:09.613 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:09.613 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:09.613 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:09.613 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:09.613 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:09.614 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:09.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:09.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:09.614 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:09.615 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:09.615 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:09.615 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:09.615 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:09.615 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:09.616 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:09.616 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:09.616 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:09.616 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:09.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:09.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:09.617 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:09.617 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:09.617 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:09.617 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:09.618 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:09.618 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:09.618 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:09.618 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:09.619 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:09.619 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:09.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:09.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:09.619 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:09.620 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:09.620 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:09.620 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:09.620 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:09.620 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:09.621 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:09.621 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:09.621 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:09.621 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:09.622 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:09.622 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:09.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:09.622 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:09.622 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:09.623 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:09.623 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:09.623 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:09.624 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T02:00:14.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:14.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:14.627 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:14.677 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:14.677 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:14.731 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:14.731 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:14.732 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:14.732 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:14.732 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:14.733 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:14.733 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:14.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:14.733 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:14.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:14.734 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:14.734 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:14.735 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:14.735 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:14.735 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:14.735 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:14.735 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:14.736 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:14.736 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:14.736 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:14.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:14.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:14.737 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:14.737 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:14.737 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:14.738 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:14.738 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:14.738 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:14.738 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:14.739 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:14.739 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:14.739 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:14.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:14.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:14.740 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:14.740 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:14.740 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:14.740 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:14.740 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:14.741 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:14.741 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:14.741 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:14.741 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:14.742 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:14.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:14.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:14.742 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:14.743 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:14.743 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:14.743 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:14.744 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:14.744 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:14.744 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:14.744 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:14.744 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:14.745 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:14.745 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:14.746 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:14.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:14.746 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:14.746 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:14.746 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:14.747 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:14.747 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:14.748 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T02:00:19.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:19.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:19.751 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:19.801 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:19.801 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:19.853 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:19.854 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:19.854 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:19.854 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:19.854 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:19.855 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:19.855 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:19.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:19.855 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:19.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:19.856 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:19.856 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:19.856 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:19.857 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:19.857 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:19.857 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:19.857 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:19.857 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:19.858 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:19.858 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:19.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:19.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:19.859 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:19.859 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:19.859 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:19.859 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:19.860 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:19.860 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:19.860 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:19.860 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:19.860 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:19.861 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:19.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:19.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:19.861 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:19.862 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:19.862 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:19.862 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:19.862 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:19.863 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:19.863 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:19.863 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:19.863 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:19.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:19.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:19.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:19.864 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:19.864 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:19.865 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:19.865 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:19.865 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:19.865 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:19.866 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:19.866 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:19.866 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:19.866 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:19.867 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:19.867 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:19.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:19.867 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:19.868 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:19.868 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:19.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:19.868 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:19.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631843906.965677_70.1 target= 2021-09-17T02:00:24.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:24.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:24.872 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:24.922 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:24.923 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:24.976 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:24.976 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:24.976 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:24.977 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:24.977 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:24.977 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:24.977 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:24.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:24.978 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:24.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:24.978 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:24.979 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:24.979 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:24.979 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631843906.965677_70.1", 2021-09-17T02:00:24.979 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:24.979 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:24.980 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:24.980 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T01:58:26.965677Z", 2021-09-17T02:00:24.980 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:24.980 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:24.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:24.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:24.981 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:24.981 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:24.982 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:24.982 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:24.982 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:24.982 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:24.982 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:24.983 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:24.983 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:24.983 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:24.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:24.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:24.984 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:24.984 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:24.984 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:24.984 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:24.985 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:24.985 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:24.985 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:24.985 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:24.986 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:24.986 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:24.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:24.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:24.987 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:24.987 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:24.987 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:24.987 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:24.988 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:24.988 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:24.988 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:24.988 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:24.988 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:24.989 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:24.989 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:24.989 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:24.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:24.990 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:24.990 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:24.990 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:24.990 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:24.991 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:24.992 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:00:24.992 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:00:24.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:24.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:24.994 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:25.045 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:25.045 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:25.098 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:00:25.099 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:00:25.099 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:00:25.099 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:00:25.099 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:00:25.100 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:25.100 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:25.100 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.100 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:00:25.101 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.101 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:00:25.101 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:25.101 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:25.101 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631843906.965677_70.1", 2021-09-17T02:00:25.102 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:25.102 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:00:25.102 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:25.102 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T01:58:26.965677Z", 2021-09-17T02:00:25.103 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:25.103 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.103 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.103 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.104 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:00:25.104 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:25.104 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:25.104 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:25.105 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:25.105 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:25.105 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:25.105 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.105 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:25.106 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.106 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.106 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.106 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:00:25.107 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:25.107 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:25.107 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:25.107 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:25.108 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:25.108 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:25.108 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.108 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:25.109 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.109 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.109 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.109 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:00:25.109 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:25.110 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:25.110 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:25.110 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:25.110 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:25.111 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:25.111 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.111 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:25.111 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.111 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.112 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:00:25.112 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.112 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:00:25.112 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:00:25.113 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:00:25.113 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.113 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:00:25.114 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:00:25.115 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:00:25.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:25.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:25.116 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:25.162 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:25.162 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:25.211 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:25.211 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.213 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631843906.965677_70.1", 2021-09-17T02:00:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:25.214 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T01:58:26.965677Z", 2021-09-17T02:00:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.216 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.217 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:25.219 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:25.221 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:25.222 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.223 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.224 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:25.225 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.226 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:25.227 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:00:25.228 INFO:rgw_multi.tests:meta checkpoint 2021-09-17T02:00:25.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:25.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:25.229 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:25.274 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:25.274 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:25.313 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:00:25.325 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:00:25.326 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.326 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631843906.965677_70.1", 2021-09-17T02:00:25.326 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T01:58:26.965677Z" 2021-09-17T02:00:25.326 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.327 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.327 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:25.327 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:25.327 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.327 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.328 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:25.328 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:25.328 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.328 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.329 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:25.329 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:25.329 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.330 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:00:25.331 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:00:25.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:25.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:25.332 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:25.376 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:25.376 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:25.426 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:25.426 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:25.427 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:25.427 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:25.427 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:25.428 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:25.428 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.428 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:25.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.429 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:25.429 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.430 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.430 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631843906.965677_70.1", 2021-09-17T02:00:25.430 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.430 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:25.431 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.431 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T01:58:26.965677Z", 2021-09-17T02:00:25.431 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.432 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.432 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:25.433 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.433 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.433 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:25.433 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.434 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:25.434 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.434 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.435 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.436 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:25.436 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.436 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.437 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:25.437 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.437 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:25.437 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.438 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.438 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.438 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.439 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:25.439 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.440 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.440 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:25.440 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.440 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:25.441 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.441 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.441 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.441 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.442 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.442 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:25.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.442 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:25.443 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:25.443 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:25.443 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.444 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:25.445 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:00:25.445 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:00:25.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:25.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:25.447 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:25.493 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:25.493 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:25.539 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:00:25.540 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:00:25.540 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:00:25.540 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:00:25.540 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:00:25.541 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:25.541 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:25.541 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.541 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:00:25.541 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.542 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:00:25.542 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:25.542 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:25.542 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631843906.965677_70.1", 2021-09-17T02:00:25.543 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:25.543 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:00:25.543 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:25.543 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T01:58:26.965677Z", 2021-09-17T02:00:25.543 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:25.544 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.544 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.544 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.544 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:00:25.544 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:25.545 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:25.545 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:25.545 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:25.545 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:25.545 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:25.546 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.546 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:25.546 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.546 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.547 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.547 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:00:25.547 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:25.547 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:25.548 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:25.548 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:25.548 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:25.548 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:25.548 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.549 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:25.549 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.549 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.549 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:25.549 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:00:25.550 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:25.550 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:25.550 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:25.550 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:25.551 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:25.551 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:25.551 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.551 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:25.551 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.552 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.552 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:00:25.552 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:25.552 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:00:25.552 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:00:25.553 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:00:25.553 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:25.553 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:00:25.554 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:00:25.554 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:00:25.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:25.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:25.556 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:25.599 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:25.646 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:25.647 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:25.647 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:25.647 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:25.647 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:25.647 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:25.648 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.648 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:25.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.648 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:25.649 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.649 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.649 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631843906.965677_70.1", 2021-09-17T02:00:25.649 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.650 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:25.650 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.650 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T01:58:26.965677Z", 2021-09-17T02:00:25.650 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.650 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.651 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:25.651 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.651 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.652 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:25.652 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.652 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:25.652 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.653 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.653 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.653 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.654 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:25.654 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.654 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.654 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:25.655 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.655 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:25.655 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.655 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.655 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.656 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:25.656 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:25.656 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:25.657 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:25.657 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:25.657 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:25.657 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:25.657 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:25.658 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:25.658 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:25.658 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.658 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.659 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:25.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:25.659 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:25.659 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:25.660 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:25.660 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:25.660 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:25.661 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:00:25.662 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph-c_2a911e49300877ce783c2becf9f484a85fa1c049/qa/../src/test/rgw/rgw_multi' with args=['tests.py', 'tests_ps.py'] 2021-09-17T02:00:25.700 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-1 2021-09-17T02:00:29.396 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-2 2021-09-17T02:00:32.541 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-3 2021-09-17T02:00:32.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:32.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:32.591 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:32.637 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:32.637 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:32.675 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:00:32.687 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:00:32.687 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:32.688 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844032.579734_82.1", 2021-09-17T02:00:32.688 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:32.579734Z" 2021-09-17T02:00:32.688 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:32.688 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:32.688 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:32.689 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:32.689 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:32.689 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:32.689 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:32.690 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:32.690 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:32.690 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:32.690 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:32.691 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:32.691 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:32.691 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:00:32.692 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:00:32.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:32.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:32.694 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:32.742 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:32.742 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:32.794 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:32.794 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:32.795 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:32.795 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:32.795 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:32.795 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:32.796 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:32.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:32.796 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:32.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:32.796 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:32.797 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:32.797 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:32.797 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844029.385420_71.1", 2021-09-17T02:00:32.797 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:32.798 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:32.798 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:32.798 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:29.385420Z", 2021-09-17T02:00:32.798 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:32.798 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:32.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:32.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:32.799 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:32.799 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:32.800 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:32.800 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:32.800 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:32.800 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:32.800 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:32.801 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:32.801 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:32.801 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:32.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:32.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:32.802 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:32.802 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:32.802 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:32.802 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:32.803 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:32.803 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:32.803 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:32.803 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:32.803 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:32.804 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:32.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:32.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:32.804 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:32.805 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:32.805 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:32.805 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:32.805 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:32.805 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:32.806 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:32.806 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:32.806 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:32.806 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:32.807 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:32.807 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:32.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:32.807 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:32.807 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:32.808 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:32.808 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:32.808 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:32.809 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631844032.579734_82.1 target=1_1631844029.385420_71.1 2021-09-17T02:00:37.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:37.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:37.812 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:37.862 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:37.916 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:37.916 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:37.917 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:37.917 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:37.917 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:37.918 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:37.918 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:37.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:37.918 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:37.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:37.919 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:37.919 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:37.919 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:37.919 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844032.579734_82.1", 2021-09-17T02:00:37.920 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:37.920 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:37.920 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:37.920 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:32.579734Z", 2021-09-17T02:00:37.921 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:37.921 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:37.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:37.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:37.922 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:37.922 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:37.922 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:37.922 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:37.923 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:37.923 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:37.923 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:37.923 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:37.923 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:37.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:37.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:37.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:37.924 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:37.925 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:37.925 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:37.925 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:37.925 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:37.925 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:37.926 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:37.926 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:37.926 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:37.926 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:37.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:37.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:37.927 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:37.927 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:37.927 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:37.928 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:37.928 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:37.928 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:37.928 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:37.929 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:37.929 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:37.929 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:37.929 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:37.929 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:37.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:37.930 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:37.930 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:37.930 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:37.931 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:37.931 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:37.932 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:00:37.933 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:00:37.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:37.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:37.934 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:37.984 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:37.985 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:38.038 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:00:38.038 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:00:38.038 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:00:38.039 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:00:38.039 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:00:38.039 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:38.039 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:38.040 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:38.040 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:00:38.040 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:38.041 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:00:38.041 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:38.041 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:38.041 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844032.579734_82.1", 2021-09-17T02:00:38.041 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:38.042 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:00:38.042 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:38.042 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:32.579734Z", 2021-09-17T02:00:38.042 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:38.043 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:38.043 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:38.043 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:38.043 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:00:38.044 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:38.044 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:38.044 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:38.044 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:38.045 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:38.045 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:38.045 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:38.045 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:38.045 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:38.046 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:38.046 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:38.046 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:00:38.046 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:38.047 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:38.047 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:38.047 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:38.047 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:38.048 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:38.048 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:38.048 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:38.048 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:38.048 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:38.049 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:38.049 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:00:38.049 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:38.049 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:38.050 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:38.050 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:38.050 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:38.050 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:38.051 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:38.051 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:38.051 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:38.051 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:38.051 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:00:38.052 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:38.052 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:00:38.052 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:00:38.053 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:00:38.053 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:38.054 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:00:38.056 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:00:38.056 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:00:38.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:38.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:38.058 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:38.107 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:38.157 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:38.158 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:38.158 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:38.158 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:38.159 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:38.159 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:38.159 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:38.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:38.160 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:38.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:38.160 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:38.161 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:38.161 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:38.161 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844032.579734_82.1", 2021-09-17T02:00:38.161 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:38.162 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:38.162 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:38.162 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:32.579734Z", 2021-09-17T02:00:38.163 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:38.163 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.163 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:38.164 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:38.164 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:38.165 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:38.165 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:38.166 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:38.166 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:38.167 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:38.167 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:38.167 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:38.168 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:38.168 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:38.169 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:38.169 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:38.169 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:38.170 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:38.170 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:38.171 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:38.171 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:38.171 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:38.172 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:38.172 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:38.172 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:38.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:38.173 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:38.173 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:38.173 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:38.174 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:38.174 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:38.174 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:38.175 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:38.175 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:38.175 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:38.175 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.176 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.176 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:38.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:38.176 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:38.177 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:38.177 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:38.177 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.177 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:38.179 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:00:38.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2021-09-17T02:00:38.200 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-4 2021-09-17T02:00:38.216 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-5 2021-09-17T02:00:38.237 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-6 2021-09-17T02:00:38.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:38.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:38.264 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:38.308 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:38.308 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:38.344 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:00:38.356 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:00:38.356 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:38.356 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844038.250008_94.1", 2021-09-17T02:00:38.357 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:38.250008Z" 2021-09-17T02:00:38.357 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:38.357 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:38.357 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:38.357 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:38.358 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:38.358 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:38.358 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:38.359 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:38.359 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:38.359 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:38.359 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:38.359 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:38.360 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:38.360 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:00:38.361 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:00:38.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:38.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:38.363 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:38.457 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:38.458 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:38.458 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:38.458 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:38.458 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:38.459 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:38.459 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:38.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:38.459 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:38.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:38.460 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:38.460 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:38.460 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:38.461 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844038.206442_83.1", 2021-09-17T02:00:38.461 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:38.461 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:38.461 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:38.461 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:38.206442Z", 2021-09-17T02:00:38.462 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:38.462 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:38.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:38.463 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:38.463 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:38.463 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:38.463 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:38.464 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:38.464 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:38.464 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:38.464 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:38.465 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:38.465 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:38.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:38.466 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:38.466 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:38.466 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:38.466 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:38.467 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:38.467 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:38.467 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:38.467 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:38.468 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:38.468 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:38.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:38.469 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:38.469 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:38.469 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:38.469 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:38.469 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:38.470 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:38.470 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:38.470 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:38.470 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:38.471 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.471 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.471 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:38.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:38.472 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:38.472 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:38.472 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:38.472 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:38.472 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:38.474 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631844038.250008_94.1 target=1_1631844038.206442_83.1 2021-09-17T02:00:43.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:43.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:43.476 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:43.526 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:43.526 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:43.584 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:43.584 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844038.250008_94.1", 2021-09-17T02:00:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:38.250008Z", 2021-09-17T02:00:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:43.592 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:43.592 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:43.592 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:43.592 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:43.592 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.597 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:43.599 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:00:43.599 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:00:43.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:43.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:43.601 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:43.650 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:43.651 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:43.704 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:00:43.705 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:00:43.705 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:00:43.705 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:00:43.705 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:00:43.705 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:43.706 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:43.706 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:43.706 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:00:43.706 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:43.707 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:00:43.707 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:43.707 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:43.707 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844038.250008_94.1", 2021-09-17T02:00:43.707 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:43.708 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:00:43.708 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:43.708 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:38.250008Z", 2021-09-17T02:00:43.708 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:43.709 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:43.709 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:43.709 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:43.709 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:00:43.709 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:43.710 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:43.710 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:43.710 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:43.710 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:43.710 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:43.711 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:43.711 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:43.711 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:43.711 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:43.712 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:43.712 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:00:43.712 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:43.712 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:43.712 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:43.713 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:43.713 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:43.713 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:43.713 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:43.713 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:43.714 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:43.714 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:43.714 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:43.714 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:00:43.715 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:43.715 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:43.715 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:43.715 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:43.715 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:43.716 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:43.716 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:43.716 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:43.716 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:43.716 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:43.717 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:00:43.717 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:43.717 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:00:43.717 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:00:43.718 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:00:43.718 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:43.718 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:00:43.719 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:00:43.720 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:00:43.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:43.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:43.721 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:43.767 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:43.768 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:43.817 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:43.817 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:43.818 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:43.818 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:43.818 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:43.818 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:43.819 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:43.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:43.819 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:43.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:43.819 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:43.820 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:43.820 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:43.820 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844038.250008_94.1", 2021-09-17T02:00:43.820 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:43.820 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:43.821 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:43.821 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:38.250008Z", 2021-09-17T02:00:43.821 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:43.821 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:43.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:43.822 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:43.822 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:43.822 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:43.823 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:43.823 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:43.823 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:43.823 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:43.823 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:43.824 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:43.824 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:43.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:43.824 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:43.825 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:43.825 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:43.825 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:43.825 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:43.826 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:43.826 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:43.826 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:43.826 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:43.826 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:43.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:43.827 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:43.827 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:43.827 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:43.828 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:43.828 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:43.828 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:43.828 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:43.829 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:43.829 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:43.829 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.829 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.829 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:43.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:43.830 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:43.830 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:43.830 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:43.830 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:43.831 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:43.832 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:00:44.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.074 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:44.120 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:44.120 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:44.157 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:00:44.169 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:00:44.170 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.170 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844038.250008_94.1", 2021-09-17T02:00:44.170 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:38.250008Z" 2021-09-17T02:00:44.170 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:44.171 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.171 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:44.171 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:44.171 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:44.171 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.172 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:44.172 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:44.172 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:44.172 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.173 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:44.173 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:44.173 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:44.173 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:00:44.174 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:00:44.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.176 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:44.222 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:44.222 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:44.273 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:44.273 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:44.274 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:44.274 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:44.274 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:44.274 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:44.275 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.275 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:44.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.275 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:44.276 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.276 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.276 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844038.250008_94.1", 2021-09-17T02:00:44.276 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.277 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:44.277 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.277 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:38.250008Z", 2021-09-17T02:00:44.277 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.277 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:44.278 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:44.279 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.281 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.282 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.283 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.284 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.285 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.286 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:44.287 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.287 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:44.288 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:00:44.289 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:00:44.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.291 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:44.334 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:44.335 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:44.381 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:00:44.381 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:00:44.381 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:00:44.382 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:00:44.382 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:00:44.382 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:44.382 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:44.382 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:44.383 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:00:44.383 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.383 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:00:44.383 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:44.384 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:44.384 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844038.250008_94.1", 2021-09-17T02:00:44.384 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:44.384 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:00:44.384 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:44.385 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:38.250008Z", 2021-09-17T02:00:44.385 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:44.385 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:44.385 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:44.386 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.386 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:00:44.386 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:44.387 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:44.387 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:44.387 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:44.387 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:44.388 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:44.388 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.388 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:44.388 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:44.389 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:44.389 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.389 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:00:44.389 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:44.389 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:44.390 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:44.390 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:44.390 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:44.390 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:44.391 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.391 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:44.391 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:44.391 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:44.391 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.392 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:00:44.392 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:44.392 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:44.392 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:44.393 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:44.393 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:44.393 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:44.393 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.393 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:44.394 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:44.394 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:44.394 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:00:44.394 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:44.395 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:00:44.395 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:00:44.395 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:00:44.395 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:44.395 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:00:44.397 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:00:44.397 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:00:44.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.399 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:44.444 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:44.445 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:44.491 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:44.491 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:44.492 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:44.492 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:44.492 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:44.492 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:44.493 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.493 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:44.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.494 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:44.494 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.494 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.494 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844038.250008_94.1", 2021-09-17T02:00:44.494 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.495 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:44.495 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.495 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:38.250008Z", 2021-09-17T02:00:44.495 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.496 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.496 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:44.496 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.497 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.497 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:44.497 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.497 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:44.498 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.498 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.498 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.498 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.499 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:44.499 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.499 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.500 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:44.500 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.500 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:44.500 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.501 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.501 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.501 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.502 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:44.502 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.502 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.502 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:44.503 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.503 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:44.503 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.503 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.504 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.504 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.504 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.504 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:44.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.505 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:44.505 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:44.505 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:44.505 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.506 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:44.507 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:00:44.525 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2021-09-17T02:00:44.526 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-7 2021-09-17T02:00:44.538 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-8 2021-09-17T02:00:44.559 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-9 2021-09-17T02:00:44.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.582 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:44.627 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:44.628 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:44.665 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:00:44.677 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:00:44.677 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.677 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844044.571786_106.1", 2021-09-17T02:00:44.677 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:44.571786Z" 2021-09-17T02:00:44.678 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:44.678 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.678 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:44.678 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:44.679 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:44.679 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.679 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:44.679 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:44.679 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:44.680 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:44.680 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:44.680 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:44.680 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:44.680 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:00:44.681 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:00:44.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:44.683 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:44.727 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:44.727 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:44.774 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:44.775 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:44.775 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:44.775 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:44.775 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:44.776 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:44.776 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.776 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:44.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.777 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:44.777 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.777 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.777 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844044.531407_95.1", 2021-09-17T02:00:44.778 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.778 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:44.778 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.778 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:44.531407Z", 2021-09-17T02:00:44.778 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.779 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.779 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:44.780 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.780 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.780 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:44.780 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.780 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:44.781 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.781 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.781 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.781 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.782 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:44.782 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.782 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.782 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:44.783 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.783 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:44.783 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.783 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.784 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.784 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:44.784 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:44.785 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:44.785 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:44.785 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:44.785 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:44.786 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:44.786 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:44.786 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:44.786 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:44.786 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.787 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.787 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:44.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:44.787 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:44.788 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:44.788 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:44.788 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:44.788 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:44.789 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631844044.571786_106.1 target=1_1631844044.531407_95.1 2021-09-17T02:00:49.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:49.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:49.792 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:49.843 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:49.897 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:49.897 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:49.897 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:49.897 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:49.898 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:49.898 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:49.898 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:49.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:49.899 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:49.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:49.899 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:49.899 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:49.900 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:49.900 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844044.571786_106.1", 2021-09-17T02:00:49.900 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:49.900 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:49.900 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:49.901 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:44.571786Z", 2021-09-17T02:00:49.901 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:49.901 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:49.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:49.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:49.902 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:49.902 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:49.902 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:49.902 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:49.902 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:49.903 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:49.903 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:49.903 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:49.903 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:49.903 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:49.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:49.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:49.904 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:49.904 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:49.904 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:49.905 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:49.905 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:49.905 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:49.905 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:49.905 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:49.906 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:49.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:49.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:49.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:49.906 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:49.907 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:49.907 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:49.907 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:49.908 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:49.908 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:49.908 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:49.908 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:49.909 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:49.909 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:49.910 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:49.910 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:49.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:49.910 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:49.911 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:49.911 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:49.911 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:49.911 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:49.912 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:00:49.913 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:00:49.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:49.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:49.915 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:49.965 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:49.965 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:50.019 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:00:50.019 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:00:50.019 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:00:50.019 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:00:50.020 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:00:50.020 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:50.020 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:50.020 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:50.021 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:00:50.021 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:50.021 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:00:50.021 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:50.022 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:50.022 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844044.571786_106.1", 2021-09-17T02:00:50.022 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:50.023 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:00:50.023 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:50.023 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:44.571786Z", 2021-09-17T02:00:50.023 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:50.023 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:50.024 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:50.024 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:50.024 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:00:50.024 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:50.025 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:50.025 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:50.025 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:50.025 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:50.025 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:50.026 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:50.026 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:50.026 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:50.026 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:50.027 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:50.027 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:00:50.027 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:50.027 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:50.028 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:50.028 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:50.028 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:50.028 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:50.028 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:50.029 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:50.029 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:50.029 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:50.029 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:50.029 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:00:50.030 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:50.030 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:50.030 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:50.030 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:50.031 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:50.031 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:50.031 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:50.031 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:50.031 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:50.032 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:50.032 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:00:50.032 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:50.032 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:00:50.033 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:00:50.033 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:00:50.033 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:50.033 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:00:50.035 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:00:50.035 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:00:50.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:50.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:50.037 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:50.083 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:50.083 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:50.133 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:50.133 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:50.134 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:50.135 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844044.571786_106.1", 2021-09-17T02:00:50.136 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:44.571786Z", 2021-09-17T02:00:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:50.137 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:50.138 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:50.139 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:50.141 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:50.142 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:50.143 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:50.144 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:50.145 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:50.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:50.147 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:50.147 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:50.147 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:50.147 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.147 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:50.149 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:00:50.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:50.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:50.283 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:50.326 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:50.327 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:50.362 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:00:50.373 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:00:50.374 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:50.374 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:00:50.374 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:00:50.375 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:50.375 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:50.375 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:50.375 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:50.376 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:50.376 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:50.376 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:50.376 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:50.377 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:50.377 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:50.377 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:50.377 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:50.377 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:50.378 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:00:50.379 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:00:50.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:50.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:50.380 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:50.426 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:50.427 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:50.474 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:50.475 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:50.475 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:50.475 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:50.476 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:50.476 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:50.476 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:50.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:50.476 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:50.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:50.477 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:50.477 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:50.477 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:50.478 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.258286_111.1", 2021-09-17T02:00:50.478 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:50.478 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:50.478 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:50.479 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.258286Z", 2021-09-17T02:00:50.479 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:50.479 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:50.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:50.480 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:50.480 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:50.480 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:50.480 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:50.481 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:50.481 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:50.481 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:50.481 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:50.482 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:50.482 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:50.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:50.482 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:50.483 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:50.483 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:50.483 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:50.483 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:50.484 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:50.484 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:50.484 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:50.484 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:50.484 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:50.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:50.485 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:50.485 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:50.485 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:50.486 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:50.486 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:50.486 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:50.487 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:50.487 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:50.487 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:50.487 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.488 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.488 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:50.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:50.488 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:50.488 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:50.489 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:50.489 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:50.489 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:50.490 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631844050.260780_112.1 target=1_1631844050.258286_111.1 2021-09-17T02:00:55.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:55.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:55.494 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:55.545 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:55.545 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:55.601 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:00:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:55.606 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:55.606 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:00:55.606 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:55.606 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:55.608 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:55.608 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:55.616 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:55.616 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.616 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.616 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.618 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:55.619 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:00:55.620 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:00:55.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:55.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:55.621 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:55.672 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:55.672 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:55.726 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:00:55.727 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:00:55.727 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:00:55.728 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:00:55.728 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:00:55.729 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:55.729 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:55.729 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:55.730 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:00:55.730 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:55.731 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:00:55.731 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:55.731 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:55.732 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:00:55.732 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:55.733 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:00:55.733 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:55.733 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:00:55.734 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:55.735 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:55.735 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:55.735 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:55.735 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:00:55.736 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:55.736 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:55.736 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:55.736 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:55.736 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:55.737 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:55.737 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:55.737 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:55.737 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:55.738 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:55.738 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:55.738 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:00:55.738 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:55.738 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:55.739 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:55.739 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:55.739 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:55.739 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:55.740 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:55.740 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:55.740 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:55.740 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:55.740 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:55.741 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:00:55.741 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:00:55.741 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:00:55.742 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:55.742 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:00:55.742 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:00:55.742 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:00:55.743 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:00:55.743 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:00:55.743 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:55.743 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:55.744 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:00:55.744 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:55.744 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:00:55.744 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:00:55.744 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:00:55.745 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:55.745 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:00:55.746 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:00:55.747 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:00:55.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:55.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:55.749 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:55.795 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:55.795 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:55.844 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:00:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:00:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:55.859 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:00:55.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2021-09-17T02:00:55.870 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-10 2021-09-17T02:00:55.886 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-11 2021-09-17T02:00:55.909 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-12 2021-09-17T02:00:56.019 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:00:56.018+0000 7f62bf631700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.019 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T02:00:56.018+0000 7f4379409700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.020 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:00:56.018+0000 7f62bfe32700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.020 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T02:00:56.018+0000 7f437a40b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.021 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T02:00:56.020+0000 7f436ebf4700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.021 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:00:56.018+0000 7f1e1c82c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.022 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:00:56.018+0000 7f1e1a027700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.217 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:00:56.216+0000 7f1e29045700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.218 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:00:56.217+0000 7f62cce4c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.219 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T02:00:56.217+0000 7f4370bf8700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.219 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:00:56.218+0000 7f62c8e44700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.252 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:00:56.251+0000 7f1e19826700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.253 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:00:56.251+0000 7f1e22037700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.253 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:00:56.252+0000 7f1e1c02b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:56.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:56.500 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:56.578 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:56.578 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:56.659 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:00:56.698 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:00:56.698 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:56.698 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:00:56.698 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:00:56.699 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:56.699 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:56.699 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844055.922434_103.1", 2021-09-17T02:00:56.699 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:55.922434Z" 2021-09-17T02:00:56.700 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:56.700 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:56.700 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:56.700 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:56.700 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:00:56.701 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:00:56.701 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:00:56.701 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:00:56.701 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:00:56.701 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:00:56.703 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:00:56.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:56.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:00:56.705 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:00:56.753 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:00:56.753 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:00:56.804 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:00:56.805 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:00:56.805 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:00:56.805 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:00:56.805 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:00:56.806 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:00:56.806 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:56.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:56.806 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:00:56.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:56.807 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:00:56.807 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:56.807 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:56.807 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:00:56.807 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:56.808 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:00:56.808 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:56.808 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:00:56.808 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:56.809 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:56.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:56.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:56.809 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:00:56.810 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:56.810 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:56.810 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:56.810 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:56.811 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:56.811 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:56.811 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:56.811 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:56.812 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:56.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:56.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:56.812 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:00:56.813 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:56.813 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:56.813 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:56.813 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:56.814 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:56.814 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:56.814 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:56.814 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:56.814 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:56.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:56.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:00:56.815 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:00:56.815 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:00:56.815 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:00:56.816 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:00:56.816 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:00:56.816 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:00:56.816 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:00:56.816 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:00:56.817 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:00:56.817 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:56.817 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:56.817 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:00:56.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:00:56.818 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:00:56.818 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:00:56.818 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:00:56.819 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:00:56.819 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:00:56.820 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844055.922434_103.1 target= 2021-09-17T02:00:56.821 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T02:00:56.817+0000 7f437c40f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:00:56.821 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T02:00:56.818+0000 7f4370bf8700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:01:01.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:01.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:01.823 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:01.875 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:01.929 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:01:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:01:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:01:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:01:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:01:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:01:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:01:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:01:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:01:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844055.877031_92.1", 2021-09-17T02:01:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:55.877031Z", 2021-09-17T02:01:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:01:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:01:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:01:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:01:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:01:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:01:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:01.944 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:01:01.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844055.922434_103.1 target=1_1631844055.877031_92.1 2021-09-17T02:01:06.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:06.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:06.948 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:06.999 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:06.999 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:07.054 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:01:07.055 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:01:07.055 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:01:07.055 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:01:07.055 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:01:07.056 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:07.056 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:07.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:07.056 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:01:07.056 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:07.057 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:01:07.057 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:07.057 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:07.057 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:07.058 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:07.058 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:01:07.058 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:07.058 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:07.059 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:07.059 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:07.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:07.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:07.059 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:01:07.060 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:07.060 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:07.060 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844055.877031_92.1", 2021-09-17T02:01:07.060 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:07.060 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:07.061 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:07.061 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:55.877031Z", 2021-09-17T02:01:07.061 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:07.061 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:07.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:07.062 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:07.062 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:01:07.062 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:07.062 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:07.063 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:07.063 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:07.063 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:07.063 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:07.064 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:07.064 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:07.065 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:07.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:07.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:07.065 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:01:07.066 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:07.066 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:07.066 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:07.067 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:07.067 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:07.067 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:07.067 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:07.068 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:07.068 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:07.068 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:07.068 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:01:07.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:07.069 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:01:07.069 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:01:07.070 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:01:07.070 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:07.070 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:01:07.072 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844055.922434_103.1 target=1_1631844055.877031_92.1 2021-09-17T02:01:12.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:12.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:12.075 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:12.126 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:12.127 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:12.180 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:01:12.181 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:01:12.181 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:01:12.181 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:01:12.182 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:01:12.182 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:12.182 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:12.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:12.183 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:01:12.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:12.183 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:01:12.183 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:12.183 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:12.184 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:12.184 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:12.184 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:01:12.184 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:12.185 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:12.185 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:12.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:12.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:12.186 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:01:12.186 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:12.186 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:12.186 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844055.922434_103.1", 2021-09-17T02:01:12.187 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:12.187 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:12.187 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:12.187 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:55.922434Z", 2021-09-17T02:01:12.187 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:12.188 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:12.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:12.188 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:01:12.189 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:12.189 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:12.189 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:12.189 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:12.189 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:12.190 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:12.190 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:12.190 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:12.190 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:12.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:12.191 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:01:12.191 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:12.192 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:12.192 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:12.192 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:12.192 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:12.193 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:12.193 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:12.193 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:12.193 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.194 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.194 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:01:12.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:12.194 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:01:12.194 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:01:12.195 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:01:12.195 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.195 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:01:12.196 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:01:12.197 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:01:12.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:12.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:12.199 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:12.249 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:12.249 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:12.308 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:01:12.308 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:01:12.309 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:01:12.309 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:01:12.309 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:01:12.309 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:12.309 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:12.310 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:12.310 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:01:12.310 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:12.310 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:01:12.311 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:12.311 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:12.311 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:12.311 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:12.311 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:01:12.312 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:12.312 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:12.312 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:12.312 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:12.313 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:12.313 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:12.313 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:01:12.313 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:12.313 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:12.314 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844055.922434_103.1", 2021-09-17T02:01:12.314 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:12.314 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:01:12.314 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:12.314 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:55.922434Z", 2021-09-17T02:01:12.315 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:12.315 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:12.315 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:12.315 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:12.316 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:01:12.316 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:12.316 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:12.316 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:01:12.316 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:12.317 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:01:12.317 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:12.317 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:01:12.317 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:12.317 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:12.318 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:12.318 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:12.318 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:01:12.318 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:12.319 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:12.319 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:01:12.319 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:12.319 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:01:12.319 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:12.320 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:01:12.320 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:12.320 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:12.320 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:12.321 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:01:12.321 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:12.321 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:01:12.321 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:01:12.322 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:01:12.322 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:12.322 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:01:12.323 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:01:12.324 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:01:12.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:12.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:12.325 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:12.372 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:12.373 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:12.423 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:01:12.423 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:01:12.424 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:01:12.424 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:01:12.424 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:01:12.424 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:12.424 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:12.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:12.425 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:01:12.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:12.425 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:01:12.426 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:12.426 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:12.426 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:12.426 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:12.426 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:01:12.427 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:12.427 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:12.427 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:12.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:12.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:12.428 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:01:12.429 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:12.429 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:12.429 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844055.922434_103.1", 2021-09-17T02:01:12.429 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:12.429 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:12.430 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:12.430 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:55.922434Z", 2021-09-17T02:01:12.430 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:12.430 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:12.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:12.431 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:01:12.431 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:12.431 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:12.432 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:12.432 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:12.432 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:12.432 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:12.433 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:12.433 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:12.433 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:12.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:12.434 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:01:12.434 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:12.434 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:12.434 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:12.435 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:12.435 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:12.435 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:12.435 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:12.435 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:12.436 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.436 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.436 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:01:12.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:12.436 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:01:12.437 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:01:12.437 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:01:12.437 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:12.437 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:01:12.439 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:01:12.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:12.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:12.440 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:12.491 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:12.508+0000 7fcbc2956380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:12.508+0000 7fcbc2956380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:12.509 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:12.508+0000 7fcbc2956380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:01:12.510 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:12.508+0000 7fcbc2956380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:12.534 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:12.534+0000 7fcbc2956380 1 bucket sync caught up with source: 2021-09-17T02:01:12.535 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000001.8.5, , 00000000001.9.5, , , , 00000000001.11.5, 00000000001.4.5, , ] 2021-09-17T02:01:12.535 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000001.8.5, , 00000000001.9.5, , , , 00000000001.11.5, 00000000001.4.5, , ] 2021-09-17T02:01:12.536 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:12.534+0000 7fcbc2956380 0 bucket checkpoint complete 2021-09-17T02:01:12.546 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-10 zones={test-zone2, test-zone1} 2021-09-17T02:01:12.552 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:12.557 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:12.557 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:12.558 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:12.558 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:12.559 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:12.603 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:12.604 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:12.604 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:12.605 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:12.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:12.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:12.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:12.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:12.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:13.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:13.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:13.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:13.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:13.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:13.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:13.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:13.670 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-10 zones={test-zone2, test-zone1} 2021-09-17T02:01:13.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:13.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:13.672 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:13.730 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:13.730 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:13.751 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:13.750+0000 7fd6a4898380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:13.751 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:13.750+0000 7fd6a4898380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:13.752 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:13.750+0000 7fd6a4898380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:01:13.752 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:13.750+0000 7fd6a4898380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:13.777 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:13.776+0000 7fd6a4898380 1 bucket sync caught up with source: 2021-09-17T02:01:13.778 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000001.8.5, , 00000000001.9.5, , , , 00000000001.11.5, 00000000001.4.5, , ] 2021-09-17T02:01:13.778 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000001.8.5, , 00000000001.9.5, , , , 00000000001.11.5, 00000000001.4.5, , ] 2021-09-17T02:01:13.778 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:13.776+0000 7fd6a4898380 0 bucket checkpoint complete 2021-09-17T02:01:13.790 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-10 zones={test-zone3, test-zone1} 2021-09-17T02:01:13.796 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:13.800 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:13.800 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:13.801 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:13.801 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:13.802 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:13.805 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:13.806 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:13.806 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:13.807 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:13.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:13.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:14.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:14.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:14.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:14.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:14.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:14.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:14.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:14.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:14.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:14.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:14.854 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-10 zones={test-zone3, test-zone1} 2021-09-17T02:01:14.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:14.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:14.856 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:14.913 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:14.914 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:14.934+0000 7ff91c1e2380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:14.934+0000 7ff91c1e2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:14.935 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:14.934+0000 7ff91c1e2380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:01:14.936 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:14.934+0000 7ff91c1e2380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:14.962 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:14.961+0000 7ff91c1e2380 1 bucket sync caught up with source: 2021-09-17T02:01:14.962 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000001.8.5, , 00000000001.9.5, , , , 00000000001.11.5, 00000000001.4.5, , ] 2021-09-17T02:01:14.963 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000001.8.5, , 00000000001.9.5, , , , 00000000001.11.5, 00000000001.4.5, , ] 2021-09-17T02:01:14.963 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:14.961+0000 7ff91c1e2380 0 bucket checkpoint complete 2021-09-17T02:01:14.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:14.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:14.976 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:15.034 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:15.034 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:15.055 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:15.054+0000 7fcc17ff7380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:15.056 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:15.054+0000 7fcc17ff7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:15.056 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:15.054+0000 7fcc17ff7380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:01:15.056 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:15.054+0000 7fcc17ff7380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:15.083 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:15.082+0000 7fcc17ff7380 1 bucket sync caught up with source: 2021-09-17T02:01:15.084 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000001.35.5, , 00000000001.12.5, , , , 00000000001.43.5, 00000000001.32.5, , ] 2021-09-17T02:01:15.084 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000001.35.5, , 00000000001.12.5, , , , 00000000001.43.5, 00000000001.32.5, , ] 2021-09-17T02:01:15.084 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:15.082+0000 7fcc17ff7380 0 bucket checkpoint complete 2021-09-17T02:01:15.097 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-11 zones={test-zone1, test-zone2} 2021-09-17T02:01:15.102 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:15.147 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:15.148 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:15.148 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:15.149 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:15.149 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:15.153 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:15.153 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:15.154 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:15.154 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:15.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:15.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:15.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:15.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:15.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:15.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:15.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:15.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:15.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:15.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:15.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:16.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:16.177 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-11 zones={test-zone1, test-zone2} 2021-09-17T02:01:16.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:16.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:16.179 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:16.236 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:16.236 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:16.258 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:16.257+0000 7fb15c4f1380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:16.258 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:16.257+0000 7fb15c4f1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:16.258 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:16.257+0000 7fb15c4f1380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:01:16.259 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:16.257+0000 7fb15c4f1380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:16.286 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:16.285+0000 7fb15c4f1380 1 bucket sync caught up with source: 2021-09-17T02:01:16.287 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000001.35.5, , 00000000001.12.5, , , , 00000000001.43.5, 00000000001.32.5, , ] 2021-09-17T02:01:16.287 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000001.35.5, , 00000000001.12.5, , , , 00000000001.43.5, 00000000001.32.5, , ] 2021-09-17T02:01:16.287 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:16.285+0000 7fb15c4f1380 0 bucket checkpoint complete 2021-09-17T02:01:16.300 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-11 zones={test-zone3, test-zone2} 2021-09-17T02:01:16.306 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:16.309 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:16.310 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:16.311 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:16.311 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:16.312 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:16.359 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:16.360 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:16.360 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:16.361 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:16.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:16.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:16.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:16.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:16.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:16.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:16.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:16.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:17.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:17.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:17.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:17.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:17.394 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-11 zones={test-zone3, test-zone2} 2021-09-17T02:01:17.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:17.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:17.396 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:17.453 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:17.454 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:17.475 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:17.473+0000 7f27f1bfc380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:17.475 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:17.473+0000 7f27f1bfc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:17.475 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:17.473+0000 7f27f1bfc380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:01:17.475 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:17.473+0000 7f27f1bfc380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:17.502 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:17.501+0000 7f27f1bfc380 1 bucket sync caught up with source: 2021-09-17T02:01:17.503 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000001.23.5, , 00000000001.124.5, , , , 00000000001.64.5, 00000000001.45.5, , ] 2021-09-17T02:01:17.503 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000001.23.5, , 00000000001.124.5, , , , 00000000001.64.5, 00000000001.45.5, , ] 2021-09-17T02:01:17.503 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:17.501+0000 7f27f1bfc380 0 bucket checkpoint complete 2021-09-17T02:01:17.516 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-12 zones={test-zone1, test-zone3} 2021-09-17T02:01:17.521 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:17.524 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:17.525 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:17.525 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:17.526 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:17.526 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:17.530 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:17.531 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:17.532 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:17.533 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:17.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:17.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:17.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:17.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:17.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:18.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:18.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:18.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:18.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:18.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:18.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:18.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:18.794 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-12 zones={test-zone1, test-zone3} 2021-09-17T02:01:18.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:18.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:18.796 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:18.853 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:18.854 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:18.875 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:18.874+0000 7fb2f3323380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:18.875 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:18.874+0000 7fb2f3323380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:18.875 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:18.874+0000 7fb2f3323380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:01:18.875 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:18.874+0000 7fb2f3323380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:18.902 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:18.902+0000 7fb2f3323380 1 bucket sync caught up with source: 2021-09-17T02:01:18.903 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000001.23.5, , 00000000001.124.5, , , , 00000000001.64.5, 00000000001.45.5, , ] 2021-09-17T02:01:18.903 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000001.23.5, , 00000000001.124.5, , , , 00000000001.64.5, 00000000001.45.5, , ] 2021-09-17T02:01:18.903 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:18.902+0000 7fb2f3323380 0 bucket checkpoint complete 2021-09-17T02:01:18.916 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-12 zones={test-zone2, test-zone3} 2021-09-17T02:01:18.921 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:18.925 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:18.925 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:18.926 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:18.926 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:18.927 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:18.971 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T02:01:18.971 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T02:01:18.972 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T02:01:18.972 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:19.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:19.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:19.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T02:01:19.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:19.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:19.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T02:01:19.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:19.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:19.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T02:01:19.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:19.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:19.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:20.054 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-12 zones={test-zone2, test-zone3} 2021-09-17T02:01:20.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2021-09-17T02:01:20.055 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-13 2021-09-17T02:01:20.072 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-14 2021-09-17T02:01:20.101 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-15 2021-09-17T02:01:20.229 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:01:20.228+0000 7f1e2503d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:01:20.230 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:01:20.229+0000 7f62bee30700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:01:20.230 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T02:01:20.229+0000 7f4375401700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:01:20.243 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:01:20.243+0000 7f62c0633700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:01:20.263 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:01:20.261+0000 7f1e2403b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:01:20.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:20.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:20.268 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:20.318 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:20.318 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:20.361 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:01:20.375 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:01:20.376 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:20.376 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:20.376 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:01:20.376 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:20.377 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:20.377 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844080.115796_125.1", 2021-09-17T02:01:20.377 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:01:20.115796Z" 2021-09-17T02:01:20.377 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:20.377 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:20.378 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:01:20.378 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:01:20.378 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:20.378 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:20.379 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:01:20.379 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:01:20.379 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:20.379 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:01:20.380 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:01:20.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:20.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:20.381 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:20.431 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:20.431 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:20.485 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:01:20.486 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:01:20.486 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:01:20.486 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:01:20.486 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:01:20.487 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:20.487 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:20.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:20.487 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:01:20.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:20.488 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:01:20.488 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:20.488 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:20.488 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:20.489 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:20.489 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:01:20.489 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:20.489 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:20.489 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:20.490 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:20.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:20.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:20.490 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:01:20.491 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:20.491 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:20.491 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844080.062367_114.1", 2021-09-17T02:01:20.491 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:20.491 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:20.492 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:20.492 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:01:20.062367Z", 2021-09-17T02:01:20.492 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:20.492 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:20.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:20.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:20.493 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:01:20.493 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:20.493 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:20.494 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:20.494 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:20.494 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:20.494 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:20.494 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:20.495 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:20.495 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:20.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:20.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:20.496 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:01:20.496 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:20.496 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:20.496 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:20.496 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:20.497 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:20.497 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:20.497 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:20.497 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:20.498 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:20.498 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:20.498 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:01:20.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:20.498 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:01:20.499 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:01:20.499 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:01:20.499 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:20.499 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:01:20.500 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844080.115796_125.1 target=1_1631844080.062367_114.1 2021-09-17T02:01:25.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:25.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:25.503 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:25.549 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:25.549 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:25.598 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:01:25.599 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:01:25.599 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:01:25.599 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:01:25.599 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:01:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:01:25.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:25.601 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:01:25.601 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:25.601 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:25.601 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:25.601 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:25.602 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:01:25.602 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:25.602 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:25.602 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:25.602 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:25.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:25.603 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:01:25.603 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:25.603 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:25.604 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844080.115796_125.1", 2021-09-17T02:01:25.604 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:25.604 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:25.604 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:25.604 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:01:20.115796Z", 2021-09-17T02:01:25.605 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:25.605 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:25.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:25.605 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:01:25.606 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:25.606 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:25.606 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:25.606 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:25.606 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:25.607 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:25.607 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:25.607 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:25.607 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:25.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:25.608 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:01:25.608 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:25.608 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:25.608 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:25.609 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:25.609 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:25.609 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:25.609 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:25.609 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:25.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.610 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:01:25.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:25.610 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:01:25.611 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:01:25.611 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:01:25.611 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.611 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:01:25.612 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:01:25.613 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:01:25.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:25.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:25.614 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:25.662 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:25.662 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:25.712 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:01:25.712 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:01:25.713 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:01:25.713 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:01:25.713 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:01:25.713 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:25.713 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:25.714 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:25.714 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:01:25.714 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:25.714 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:01:25.715 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:25.715 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:25.715 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:25.715 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:25.715 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:01:25.716 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:25.716 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:25.716 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:25.716 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:25.716 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:25.717 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:25.717 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:01:25.717 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:25.717 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:25.717 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844080.115796_125.1", 2021-09-17T02:01:25.718 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:25.718 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:01:25.718 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:25.718 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:01:20.115796Z", 2021-09-17T02:01:25.718 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:25.719 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:25.719 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:25.719 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:25.719 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:01:25.719 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:25.720 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:25.720 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:01:25.720 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:25.720 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:01:25.721 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:25.721 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:01:25.721 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:25.721 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:25.721 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:25.722 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:25.722 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:01:25.722 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:25.722 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:25.723 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:01:25.723 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:25.723 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:01:25.723 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:25.723 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:01:25.724 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:25.724 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:25.724 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:25.724 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:01:25.724 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:25.725 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:01:25.725 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:01:25.725 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:01:25.725 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:25.725 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:01:25.727 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:01:25.727 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:01:25.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:25.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:25.729 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:25.772 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:25.772 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:25.818 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:01:25.819 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:01:25.819 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:01:25.819 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:01:25.819 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:01:25.820 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:25.820 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:25.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:25.820 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:01:25.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:25.821 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:01:25.821 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:25.822 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:25.822 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:25.822 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:25.822 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:01:25.823 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:25.823 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:25.823 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:25.823 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:25.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:25.824 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:01:25.824 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:25.824 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:25.825 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844080.115796_125.1", 2021-09-17T02:01:25.825 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:25.825 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:25.825 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:25.825 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:01:20.115796Z", 2021-09-17T02:01:25.826 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:25.826 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:25.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:25.827 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:01:25.827 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:25.827 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:25.827 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:25.827 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:25.828 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:25.828 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:25.828 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:25.828 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:25.829 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:25.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:25.829 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:01:25.830 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:25.830 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:25.830 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:25.831 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:25.831 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:25.831 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:25.831 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:25.831 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:25.832 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.832 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.832 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:01:25.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:25.833 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:01:25.833 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:01:25.833 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:01:25.833 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:25.834 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:01:25.835 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:01:25.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:25.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:25.837 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:25.926 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:25.927 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:25.945 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:25.943+0000 7f8c67644380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:25.945 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:25.943+0000 7f8c67644380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:25.945 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:25.943+0000 7f8c67644380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:01:25.945 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:25.943+0000 7f8c67644380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:25.969 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:25.967+0000 7f8c67644380 1 bucket sync caught up with source: 2021-09-17T02:01:25.969 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2021-09-17T02:01:25.969 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2021-09-17T02:01:25.970 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:25.967+0000 7f8c67644380 0 bucket checkpoint complete 2021-09-17T02:01:25.981 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-13 zones={test-zone2, test-zone1} 2021-09-17T02:01:25.987 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:26.032 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:26.035 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:26.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:26.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:26.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:26.370 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-13 zones={test-zone2, test-zone1} 2021-09-17T02:01:26.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:26.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:26.372 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:26.428 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:26.428 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:26.449 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:26.448+0000 7fccc5c4e380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:26.450 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:26.448+0000 7fccc5c4e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:26.450 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:26.448+0000 7fccc5c4e380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:01:26.450 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:26.448+0000 7fccc5c4e380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:26.476 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:26.474+0000 7fccc5c4e380 1 bucket sync caught up with source: 2021-09-17T02:01:26.476 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2021-09-17T02:01:26.476 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2021-09-17T02:01:26.477 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:26.474+0000 7fccc5c4e380 0 bucket checkpoint complete 2021-09-17T02:01:26.488 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-13 zones={test-zone3, test-zone1} 2021-09-17T02:01:26.494 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:26.539 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:26.539 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:26.543 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:26.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:26.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:26.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:26.846 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-13 zones={test-zone3, test-zone1} 2021-09-17T02:01:26.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:26.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:26.848 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:26.906 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:26.907 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:26.927 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:26.926+0000 7f9e5fa04380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:26.926+0000 7f9e5fa04380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:26.926+0000 7f9e5fa04380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:01:26.928 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:26.926+0000 7f9e5fa04380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:26.955 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:26.953+0000 7f9e5fa04380 1 bucket sync caught up with source: 2021-09-17T02:01:26.955 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2021-09-17T02:01:26.955 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2021-09-17T02:01:26.955 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:26.953+0000 7f9e5fa04380 0 bucket checkpoint complete 2021-09-17T02:01:26.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:26.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:26.969 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:27.026 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:27.026 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:27.047 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.046+0000 7f351f314380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:27.048 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.046+0000 7f351f314380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:27.048 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.046+0000 7f351f314380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:01:27.048 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.046+0000 7f351f314380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:27.075 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.073+0000 7f351f314380 1 bucket sync caught up with source: 2021-09-17T02:01:27.075 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000001.126.5, , , , , , , , , ] 2021-09-17T02:01:27.075 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000001.126.5, , , , , , , , , ] 2021-09-17T02:01:27.076 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.073+0000 7f351f314380 0 bucket checkpoint complete 2021-09-17T02:01:27.089 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-14 zones={test-zone1, test-zone2} 2021-09-17T02:01:27.094 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:27.139 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:27.140 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:27.143 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:27.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:27.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:27.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:27.446 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-14 zones={test-zone1, test-zone2} 2021-09-17T02:01:27.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:27.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:27.448 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:27.504 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:27.505 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:27.526 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.525+0000 7f015f35a380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:27.527 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.525+0000 7f015f35a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:27.527 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.525+0000 7f015f35a380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:01:27.527 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.525+0000 7f015f35a380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:27.553 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.552+0000 7f015f35a380 1 bucket sync caught up with source: 2021-09-17T02:01:27.553 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000001.126.5, , , , , , , , , ] 2021-09-17T02:01:27.554 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000001.126.5, , , , , , , , , ] 2021-09-17T02:01:27.554 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.552+0000 7f015f35a380 0 bucket checkpoint complete 2021-09-17T02:01:27.567 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-14 zones={test-zone3, test-zone2} 2021-09-17T02:01:27.572 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:27.619 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:27.619 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:27.623 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:27.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:27.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:27.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:27.922 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-14 zones={test-zone3, test-zone2} 2021-09-17T02:01:27.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:27.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:27.924 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:27.977 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:27.978 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:27.997 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.996+0000 7eff32ec7380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:27.998 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.996+0000 7eff32ec7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:27.998 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.996+0000 7eff32ec7380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:01:27.998 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:27.996+0000 7eff32ec7380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:28.023 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:28.022+0000 7eff32ec7380 1 bucket sync caught up with source: 2021-09-17T02:01:28.023 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000001.199.5, , , , , , , , , ] 2021-09-17T02:01:28.023 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000001.199.5, , , , , , , , , ] 2021-09-17T02:01:28.024 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:28.022+0000 7eff32ec7380 0 bucket checkpoint complete 2021-09-17T02:01:28.036 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-15 zones={test-zone1, test-zone3} 2021-09-17T02:01:28.043 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:28.096 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:28.097 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:28.108 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:28.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:28.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:28.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:28.463 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-15 zones={test-zone1, test-zone3} 2021-09-17T02:01:28.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:28.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:28.465 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:28.523 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:28.544 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:28.543+0000 7f7d7b0ea380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:28.543+0000 7f7d7b0ea380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:28.543+0000 7f7d7b0ea380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:01:28.545 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:28.543+0000 7f7d7b0ea380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:28.572 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:28.571+0000 7f7d7b0ea380 1 bucket sync caught up with source: 2021-09-17T02:01:28.573 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000001.199.5, , , , , , , , , ] 2021-09-17T02:01:28.573 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000001.199.5, , , , , , , , , ] 2021-09-17T02:01:28.573 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:28.571+0000 7f7d7b0ea380 0 bucket checkpoint complete 2021-09-17T02:01:28.585 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-15 zones={test-zone2, test-zone3} 2021-09-17T02:01:28.590 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:28.639 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:28.640 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:28.643 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T02:01:28.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:28.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:28.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T02:01:28.938 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-15 zones={test-zone2, test-zone3} 2021-09-17T02:01:28.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:28.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:28.948 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:29.003 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:29.023 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:29.021+0000 7f9361da7380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:29.023 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:29.021+0000 7f9361da7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:29.024 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:29.021+0000 7f9361da7380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:01:29.024 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:29.021+0000 7f9361da7380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:29.050 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:29.049+0000 7f9361da7380 1 waiting for incremental sync to catch up: 2021-09-17T02:01:29.051 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2021-09-17T02:01:29.051 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2021-09-17T02:01:34.052 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:34.050+0000 7f9361da7380 1 bucket sync caught up with source: 2021-09-17T02:01:34.052 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2021-09-17T02:01:34.053 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2021-09-17T02:01:34.053 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:34.050+0000 7f9361da7380 0 bucket checkpoint complete 2021-09-17T02:01:34.068 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-13 zones={test-zone2, test-zone1} 2021-09-17T02:01:34.074 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:34.119 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:34.170 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-13 zones={test-zone2, test-zone1} 2021-09-17T02:01:34.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:34.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:34.172 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:34.228 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:34.229 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:34.249 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:34.248+0000 7f8d09f83380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:34.250 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:34.248+0000 7f8d09f83380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:34.250 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:34.248+0000 7f8d09f83380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:01:34.250 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:34.248+0000 7f8d09f83380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:34.279 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:34.278+0000 7f8d09f83380 1 bucket sync caught up with source: 2021-09-17T02:01:34.280 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2021-09-17T02:01:34.280 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2021-09-17T02:01:34.280 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:34.278+0000 7f8d09f83380 0 bucket checkpoint complete 2021-09-17T02:01:34.292 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-13 zones={test-zone3, test-zone1} 2021-09-17T02:01:34.298 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:34.343 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:34.394 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-13 zones={test-zone3, test-zone1} 2021-09-17T02:01:34.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:34.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:34.395 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:34.452 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:34.452 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:34.473 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:34.472+0000 7f0d1a112380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:34.474 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:34.472+0000 7f0d1a112380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:34.474 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:34.472+0000 7f0d1a112380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:01:34.474 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:34.472+0000 7f0d1a112380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:34.500 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:34.499+0000 7f0d1a112380 1 bucket sync caught up with source: 2021-09-17T02:01:34.500 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000002.13.5, , , , , , , , , ] 2021-09-17T02:01:34.501 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000002.13.5, , , , , , , , , ] 2021-09-17T02:01:34.501 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:34.499+0000 7f0d1a112380 0 bucket checkpoint complete 2021-09-17T02:01:34.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:34.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:34.523 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:34.576 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:34.577 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:34.596 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:34.594+0000 7f5babe17380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:34.596 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:34.594+0000 7f5babe17380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:34.596 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:34.594+0000 7f5babe17380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:01:34.596 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:34.594+0000 7f5babe17380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:34.621 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:34.619+0000 7f5babe17380 1 waiting for incremental sync to catch up: 2021-09-17T02:01:34.621 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000001.126.5, , , , , , , , , ] 2021-09-17T02:01:34.621 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000002.162.5, , , , , , , , , ] 2021-09-17T02:01:39.622 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:39.621+0000 7f5babe17380 1 bucket sync caught up with source: 2021-09-17T02:01:39.623 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000002.162.5, , , , , , , , , ] 2021-09-17T02:01:39.623 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000002.162.5, , , , , , , , , ] 2021-09-17T02:01:39.623 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:39.621+0000 7f5babe17380 0 bucket checkpoint complete 2021-09-17T02:01:39.639 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-14 zones={test-zone1, test-zone2} 2021-09-17T02:01:39.645 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:39.695 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:39.746 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-14 zones={test-zone1, test-zone2} 2021-09-17T02:01:39.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:39.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:39.747 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:39.806 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:39.807 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:39.827 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:39.826+0000 7f2b16f22380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:39.827 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:39.826+0000 7f2b16f22380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:39.828 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:39.826+0000 7f2b16f22380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:01:39.828 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:39.826+0000 7f2b16f22380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:39.854 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:39.853+0000 7f2b16f22380 1 bucket sync caught up with source: 2021-09-17T02:01:39.855 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000002.162.5, , , , , , , , , ] 2021-09-17T02:01:39.855 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000002.162.5, , , , , , , , , ] 2021-09-17T02:01:39.855 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:39.853+0000 7f2b16f22380 0 bucket checkpoint complete 2021-09-17T02:01:39.868 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-14 zones={test-zone3, test-zone2} 2021-09-17T02:01:39.873 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:39.919 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:39.974 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-14 zones={test-zone3, test-zone2} 2021-09-17T02:01:39.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:39.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:39.983 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:40.035 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:40.036 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:40.055 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:40.053+0000 7fee02d22380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:40.055 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:40.053+0000 7fee02d22380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:40.056 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:40.053+0000 7fee02d22380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:01:40.056 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:40.053+0000 7fee02d22380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:40.079 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:40.078+0000 7fee02d22380 1 waiting for incremental sync to catch up: 2021-09-17T02:01:40.080 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000001.199.5, , , , , , , , , ] 2021-09-17T02:01:40.080 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000002.241.5, , , , , , , , , ] 2021-09-17T02:01:45.081 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:45.079+0000 7fee02d22380 1 bucket sync caught up with source: 2021-09-17T02:01:45.081 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000002.241.5, , , , , , , , , ] 2021-09-17T02:01:45.082 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000002.241.5, , , , , , , , , ] 2021-09-17T02:01:45.082 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:45.079+0000 7fee02d22380 0 bucket checkpoint complete 2021-09-17T02:01:45.097 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-15 zones={test-zone1, test-zone3} 2021-09-17T02:01:45.103 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:45.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:45.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-15 zones={test-zone1, test-zone3} 2021-09-17T02:01:45.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:45.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:45.207 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:45.266 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:45.266 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:45.288 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:45.286+0000 7f4239601380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:45.288 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:45.286+0000 7f4239601380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:45.289 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:45.286+0000 7f4239601380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:01:45.289 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:45.286+0000 7f4239601380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:45.315 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:45.313+0000 7f4239601380 1 bucket sync caught up with source: 2021-09-17T02:01:45.316 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000002.241.5, , , , , , , , , ] 2021-09-17T02:01:45.316 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000002.241.5, , , , , , , , , ] 2021-09-17T02:01:45.316 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:45.313+0000 7f4239601380 0 bucket checkpoint complete 2021-09-17T02:01:45.328 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-15 zones={test-zone2, test-zone3} 2021-09-17T02:01:45.334 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:01:45.383 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:01:45.438 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-15 zones={test-zone2, test-zone3} 2021-09-17T02:01:45.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2021-09-17T02:01:45.439 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-16 2021-09-17T02:01:45.464 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-17 2021-09-17T02:01:45.490 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-18 2021-09-17T02:01:45.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:45.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:45.541 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:45.585 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:45.586 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:45.622 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:01:45.634 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:01:45.634 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:45.635 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:45.635 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:01:45.635 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:45.635 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:45.635 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844105.536699_149.1", 2021-09-17T02:01:45.636 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:01:45.536699Z" 2021-09-17T02:01:45.636 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:45.636 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:45.636 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:01:45.636 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:01:45.637 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:45.637 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:45.637 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:01:45.637 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:01:45.638 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:45.638 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:01:45.639 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:01:45.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:45.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:45.641 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:45.687 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:45.688 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:45.737 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:01:45.738 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:01:45.738 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:01:45.738 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:01:45.738 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:01:45.739 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:45.739 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:45.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:45.739 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:01:45.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:45.740 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:01:45.740 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:45.740 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:45.740 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:45.741 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:45.741 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:01:45.741 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:45.741 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:45.742 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:45.742 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:45.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:45.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:45.742 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:01:45.743 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:45.743 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:45.743 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844105.448058_132.1", 2021-09-17T02:01:45.743 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:45.744 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:45.744 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:45.744 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:01:45.448058Z", 2021-09-17T02:01:45.744 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:45.744 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:45.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:45.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:45.745 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:01:45.745 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:45.746 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:45.746 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:45.746 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:45.746 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:45.746 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:45.747 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:45.747 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:45.747 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:45.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:45.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:45.748 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:01:45.748 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:45.748 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:45.748 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:45.749 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:45.749 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:45.749 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:45.749 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:45.749 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:45.750 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:45.750 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:45.750 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:01:45.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:45.751 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:01:45.751 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:01:45.751 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:01:45.751 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:45.751 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:01:45.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844105.536699_149.1 target=1_1631844105.448058_132.1 2021-09-17T02:01:50.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:50.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:50.756 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:50.806 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:50.807 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:50.861 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:01:50.861 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:01:50.861 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:01:50.862 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:01:50.862 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:01:50.862 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:50.862 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:50.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:50.863 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:01:50.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:50.863 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:01:50.864 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:50.864 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:50.864 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:50.864 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:50.864 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:01:50.865 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:50.865 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:50.865 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:50.865 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:50.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:50.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:50.866 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:01:50.866 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:50.867 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:50.867 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844105.536699_149.1", 2021-09-17T02:01:50.867 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:50.867 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:50.868 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:50.868 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:01:45.536699Z", 2021-09-17T02:01:50.868 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:50.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:50.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:50.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:50.869 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:01:50.869 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:50.869 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:50.870 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:50.870 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:50.870 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:50.870 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:50.871 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:50.871 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:50.871 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:50.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:50.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:50.872 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:01:50.872 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:50.872 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:50.872 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:50.873 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:50.873 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:50.873 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:50.873 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:50.874 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:50.874 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:50.874 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:50.874 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:01:50.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:50.875 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:01:50.875 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:01:50.875 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:01:50.876 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:50.876 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:01:50.877 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:01:50.878 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:01:50.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:50.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:50.880 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:50.930 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:50.930 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:50.984 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:01:50.984 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:01:50.985 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:01:50.985 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:01:50.985 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:01:50.986 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:50.986 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:50.986 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:50.986 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:01:50.987 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:50.987 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:01:50.987 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:50.987 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:50.987 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:50.988 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:50.988 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:01:50.988 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:50.988 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:50.989 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:50.989 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:50.989 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:50.989 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:50.989 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:01:50.990 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:50.990 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:50.990 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844105.536699_149.1", 2021-09-17T02:01:50.990 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:50.991 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:01:50.991 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:50.991 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:01:45.536699Z", 2021-09-17T02:01:50.991 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:50.991 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:50.992 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:50.992 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:50.992 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:01:50.992 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:50.993 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:50.993 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:01:50.993 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:50.993 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:01:50.993 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:50.994 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:01:50.994 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:50.994 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:50.994 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:50.995 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:01:50.995 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:01:50.995 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:01:50.995 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:01:50.995 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:01:50.996 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:01:50.996 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:01:50.996 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:01:50.996 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:01:50.997 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:01:50.997 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:50.997 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:50.997 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:01:50.997 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:01:50.998 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:01:50.998 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:01:50.998 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:01:50.998 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:01:50.998 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:01:51.000 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:01:51.000 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:01:51.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:51.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:01:51.002 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:01:51.048 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:51.048 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:51.097 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:01:51.098 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:01:51.098 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:01:51.098 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:01:51.099 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:01:51.099 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:01:51.099 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:51.099 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:51.099 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:01:51.100 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:51.100 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:01:51.100 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:51.100 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:51.101 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:01:51.101 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:51.101 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:01:51.102 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:51.102 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:01:51.102 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:51.102 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:51.103 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:51.103 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:51.103 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:01:51.103 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:51.103 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:51.104 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844105.536699_149.1", 2021-09-17T02:01:51.104 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:51.104 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:51.104 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:51.105 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:01:45.536699Z", 2021-09-17T02:01:51.105 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:51.105 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:51.105 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:51.105 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:51.106 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:01:51.106 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:51.106 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:51.106 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:51.107 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:51.107 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:51.107 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:51.107 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:51.108 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:51.108 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:51.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:51.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:01:51.108 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:01:51.109 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:01:51.109 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:01:51.109 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:01:51.109 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:01:51.110 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:01:51.110 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:01:51.110 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:01:51.110 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:01:51.110 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:51.111 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:51.111 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:01:51.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:01:51.111 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:01:51.112 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:01:51.112 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:01:51.112 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:01:51.112 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:01:51.114 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:01:51.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:51.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:51.162 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:51.213 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:51.214 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:51.231 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:51.230+0000 7f30efd6e380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:51.232 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:51.230+0000 7f30efd6e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:51.232 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:51.230+0000 7f30efd6e380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:01:51.232 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:51.230+0000 7f30efd6e380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:51.256 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:51.255+0000 7f30efd6e380 1 waiting for incremental sync to catch up: 2021-09-17T02:01:51.257 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , , , ] 2021-09-17T02:01:51.257 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , 00000000002.10.10, , , , , ] 2021-09-17T02:01:56.258 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:56.257+0000 7f30efd6e380 1 bucket sync caught up with source: 2021-09-17T02:01:56.258 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , 00000000006.18.9, , , , , ] 2021-09-17T02:01:56.259 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , 00000000002.10.10, , , , , ] 2021-09-17T02:01:56.259 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:56.257+0000 7f30efd6e380 0 bucket checkpoint complete 2021-09-17T02:01:56.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:56.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:56.277 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:56.336 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:56.336 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:56.356 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:56.355+0000 7ff475bb9380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:56.357 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:56.355+0000 7ff475bb9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:56.357 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:56.355+0000 7ff475bb9380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:01:56.357 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:56.355+0000 7ff475bb9380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:56.383 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:56.382+0000 7ff475bb9380 1 bucket sync caught up with source: 2021-09-17T02:01:56.383 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000006.18.9, , , , , ] 2021-09-17T02:01:56.384 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000006.18.9, , , , , ] 2021-09-17T02:01:56.384 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:56.382+0000 7ff475bb9380 0 bucket checkpoint complete 2021-09-17T02:01:56.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:56.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:56.398 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:56.455 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:56.456 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:56.476+0000 7f6416858380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:56.476+0000 7f6416858380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:56.476+0000 7f6416858380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:01:56.477 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:56.476+0000 7f6416858380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:56.504 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:56.504+0000 7f6416858380 1 bucket sync caught up with source: 2021-09-17T02:01:56.505 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , 00000000006.18.9, , , , , ] 2021-09-17T02:01:56.505 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , 00000000006.18.9, , , , , ] 2021-09-17T02:01:56.505 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:01:56.504+0000 7f6416858380 0 bucket checkpoint complete 2021-09-17T02:01:56.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:56.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:01:56.573 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:01:56.626 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:01:56.627 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:01:56.646 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:56.645+0000 7fb73aeb3380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:01:56.647 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:56.645+0000 7fb73aeb3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:01:56.647 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:56.645+0000 7fb73aeb3380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:01:56.647 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:56.645+0000 7fb73aeb3380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:01:56.673 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:01:56.671+0000 7fb73aeb3380 1 waiting for incremental sync to catch up: 2021-09-17T02:01:56.673 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , , , ] 2021-09-17T02:01:56.673 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000002.580.10, , , , , ] 2021-09-17T02:02:01.674 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.674+0000 7fb73aeb3380 1 bucket sync caught up with source: 2021-09-17T02:02:01.675 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000006.588.9, , , , , ] 2021-09-17T02:02:01.675 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000002.580.10, , , , , ] 2021-09-17T02:02:01.676 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.674+0000 7fb73aeb3380 0 bucket checkpoint complete 2021-09-17T02:02:01.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:01.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:01.693 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:01.748 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:01.749 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:01.770 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.769+0000 7f3344486380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:01.770 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.769+0000 7f3344486380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:01.770 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.769+0000 7f3344486380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:01.770 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.769+0000 7f3344486380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:01.796 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.796+0000 7f3344486380 1 bucket sync caught up with source: 2021-09-17T02:02:01.796 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000006.588.9, , , , , ] 2021-09-17T02:02:01.797 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000006.588.9, , , , , ] 2021-09-17T02:02:01.797 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.796+0000 7f3344486380 0 bucket checkpoint complete 2021-09-17T02:02:01.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:01.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:01.856 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:01.909 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:01.909 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:01.929 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.929+0000 7fac2d310380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:01.930 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.929+0000 7fac2d310380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:01.930 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.929+0000 7fac2d310380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:01.930 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.929+0000 7fac2d310380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:01.955 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:01.954+0000 7fac2d310380 1 waiting for incremental sync to catch up: 2021-09-17T02:02:01.955 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , , , ] 2021-09-17T02:02:01.956 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000002.43.10, , , , , ] 2021-09-17T02:02:06.956 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:06.956+0000 7fac2d310380 1 bucket sync caught up with source: 2021-09-17T02:02:06.957 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000006.51.9, , , , , ] 2021-09-17T02:02:06.957 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000002.43.10, , , , , ] 2021-09-17T02:02:06.957 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:06.956+0000 7fac2d310380 0 bucket checkpoint complete 2021-09-17T02:02:06.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:06.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:06.974 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:07.053 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:07.053+0000 7fb693873380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:07.054 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:07.053+0000 7fb693873380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:07.054 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:07.053+0000 7fb693873380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:07.054 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:07.053+0000 7fb693873380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:07.081 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:07.080+0000 7fb693873380 1 bucket sync caught up with source: 2021-09-17T02:02:07.081 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , 00000000006.51.9, , , , , ] 2021-09-17T02:02:07.081 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , 00000000006.51.9, , , , , ] 2021-09-17T02:02:07.082 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:07.080+0000 7fb693873380 0 bucket checkpoint complete 2021-09-17T02:02:08.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:08.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:08.501 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:08.553 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:08.554 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:08.574 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.572+0000 7f4d76dce380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:08.574 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.572+0000 7f4d76dce380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:08.574 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.572+0000 7f4d76dce380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:08.575 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.572+0000 7f4d76dce380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:08.598 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.598+0000 7f4d76dce380 1 bucket sync caught up with source: 2021-09-17T02:02:08.599 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000010.26.5, , 00000000006.22.5, , 00000000012.32.5, , , , , ] 2021-09-17T02:02:08.599 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000010.26.5, , 00000000006.22.5, , 00000000012.32.5, , , , , ] 2021-09-17T02:02:08.599 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.598+0000 7f4d76dce380 0 bucket checkpoint complete 2021-09-17T02:02:08.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:08.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:08.613 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:08.667 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:08.667 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:08.686 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.686+0000 7fde29bf1380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:08.687 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.686+0000 7fde29bf1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:08.687 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.686+0000 7fde29bf1380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:08.687 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.686+0000 7fde29bf1380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:08.716 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.715+0000 7fde29bf1380 1 bucket sync caught up with source: 2021-09-17T02:02:08.716 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000010.26.5, , 00000000006.22.5, , 00000000012.32.5, , , , , ] 2021-09-17T02:02:08.716 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000010.26.5, , 00000000006.22.5, , 00000000012.32.5, , , , , ] 2021-09-17T02:02:08.717 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.715+0000 7fde29bf1380 0 bucket checkpoint complete 2021-09-17T02:02:08.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:08.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:08.729 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:08.779 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:08.797 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.796+0000 7ffabcdd8380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:08.797 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.796+0000 7ffabcdd8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:08.797 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.796+0000 7ffabcdd8380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:02:08.798 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.796+0000 7ffabcdd8380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:08.822 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.821+0000 7ffabcdd8380 1 bucket sync caught up with source: 2021-09-17T02:02:08.822 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000010.26.5, , 00000000006.22.5, , 00000000012.32.5, , , , , ] 2021-09-17T02:02:08.822 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000010.26.5, , 00000000006.22.5, , 00000000012.32.5, , , , , ] 2021-09-17T02:02:08.822 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:08.821+0000 7ffabcdd8380 0 bucket checkpoint complete 2021-09-17T02:02:08.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:08.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:08.835 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:08.886 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:08.887 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:08.906 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.905+0000 7f822159b380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:08.906 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.905+0000 7f822159b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:08.906 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.905+0000 7f822159b380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:08.906 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.905+0000 7f822159b380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:08.930 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.930+0000 7f822159b380 1 bucket sync caught up with source: 2021-09-17T02:02:08.930 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000003.218.9, , 00000000010.282.5, , 00000000010.240.5, , , , , ] 2021-09-17T02:02:08.931 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000003.218.9, , 00000000010.282.5, , 00000000010.240.5, , , , , ] 2021-09-17T02:02:08.931 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:08.930+0000 7f822159b380 0 bucket checkpoint complete 2021-09-17T02:02:08.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:08.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:08.944 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:08.995 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:08.995 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:09.013 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.013+0000 7fc6c282a380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:09.013 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.013+0000 7fc6c282a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:09.013 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.013+0000 7fc6c282a380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:09.014 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.013+0000 7fc6c282a380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:09.046 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.037+0000 7fc6c282a380 1 bucket sync caught up with source: 2021-09-17T02:02:09.047 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000003.218.9, , 00000000010.282.5, , 00000000010.240.5, , , , , ] 2021-09-17T02:02:09.047 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000003.218.9, , 00000000010.282.5, , 00000000010.240.5, , , , , ] 2021-09-17T02:02:09.047 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.037+0000 7fc6c282a380 0 bucket checkpoint complete 2021-09-17T02:02:09.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:09.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:09.051 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:09.101 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:09.101 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:09.119 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.119+0000 7fd26ca98380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:09.119 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.119+0000 7fd26ca98380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:09.120 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.119+0000 7fd26ca98380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:09.120 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.119+0000 7fd26ca98380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:09.144 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.144+0000 7fd26ca98380 1 bucket sync caught up with source: 2021-09-17T02:02:09.144 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000003.236.9, , 00000000006.213.5, , 00000000014.220.5, , , , , ] 2021-09-17T02:02:09.144 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000003.236.9, , 00000000006.213.5, , 00000000014.220.5, , , , , ] 2021-09-17T02:02:09.145 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:09.144+0000 7fd26ca98380 0 bucket checkpoint complete 2021-09-17T02:02:09.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:09.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:09.158 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:09.210 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:09.210 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:09.228 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:09.228+0000 7f35dd96a380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:09.228+0000 7f35dd96a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:09.228+0000 7f35dd96a380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:09.229 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:09.228+0000 7f35dd96a380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:09.253 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:09.252+0000 7f35dd96a380 1 bucket sync caught up with source: 2021-09-17T02:02:09.253 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000003.236.9, , 00000000006.213.5, , 00000000014.220.5, , , , , ] 2021-09-17T02:02:09.254 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000003.236.9, , 00000000006.213.5, , 00000000014.220.5, , , , , ] 2021-09-17T02:02:09.254 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:09.252+0000 7f35dd96a380 0 bucket checkpoint complete 2021-09-17T02:02:09.265 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-16 zones={test-zone2, test-zone1} 2021-09-17T02:02:09.271 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:09.274 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:09.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:09.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:09.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:09.276 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:09.279 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:09.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:09.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:09.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:09.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:09.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:09.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:09.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:09.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:09.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:09.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:09.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:09.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:09.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:09.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:10.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:10.142 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-16 zones={test-zone2, test-zone1} 2021-09-17T02:02:10.142 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-16 zones={test-zone3, test-zone1} 2021-09-17T02:02:10.148 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:10.151 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:10.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:10.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:10.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:10.153 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:10.157 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:10.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:10.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:10.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:10.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:10.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:10.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:10.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:10.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:10.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:10.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:10.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:10.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:10.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:10.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:10.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:11.070 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-16 zones={test-zone3, test-zone1} 2021-09-17T02:02:11.071 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-16 zones={test-zone3, test-zone2} 2021-09-17T02:02:11.076 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:11.080 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:11.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:11.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:11.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:11.082 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:11.127 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:11.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:11.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:11.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:11.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:11.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:11.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:11.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:11.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:11.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:11.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:11.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:11.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:11.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:11.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:11.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:12.070 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-16 zones={test-zone3, test-zone2} 2021-09-17T02:02:12.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.072 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:12.130 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:12.130 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:12.151 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.150+0000 7f9e70a5d380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:12.151 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.150+0000 7f9e70a5d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:12.151 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.150+0000 7f9e70a5d380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:12.151 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.150+0000 7f9e70a5d380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:12.178 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.177+0000 7f9e70a5d380 1 bucket sync caught up with source: 2021-09-17T02:02:12.178 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000010.21.5, , 00000000003.13.9, , 00000000007.26.9, , , , , ] 2021-09-17T02:02:12.178 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000010.21.5, , 00000000003.13.9, , 00000000007.26.9, , , , , ] 2021-09-17T02:02:12.178 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.177+0000 7f9e70a5d380 0 bucket checkpoint complete 2021-09-17T02:02:12.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.192 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:12.250 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:12.250 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:12.273 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.272+0000 7f2da06b4380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:12.273 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.272+0000 7f2da06b4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:12.273 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.272+0000 7f2da06b4380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:12.274 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.272+0000 7f2da06b4380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:12.300 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.300+0000 7f2da06b4380 1 bucket sync caught up with source: 2021-09-17T02:02:12.301 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000010.21.5, , 00000000003.13.9, , 00000000007.26.9, , , , , ] 2021-09-17T02:02:12.301 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000010.21.5, , 00000000003.13.9, , 00000000007.26.9, , , , , ] 2021-09-17T02:02:12.301 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.300+0000 7f2da06b4380 0 bucket checkpoint complete 2021-09-17T02:02:12.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.315 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:12.369 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:12.369 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:12.389 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.388+0000 7fd54976b380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:12.389 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.388+0000 7fd54976b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:12.390 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.388+0000 7fd54976b380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:02:12.390 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.388+0000 7fd54976b380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:12.415 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.414+0000 7fd54976b380 1 bucket sync caught up with source: 2021-09-17T02:02:12.415 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000010.21.5, , 00000000003.13.9, , 00000000007.26.9, , , , , ] 2021-09-17T02:02:12.415 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000010.21.5, , 00000000003.13.9, , 00000000007.26.9, , , , , ] 2021-09-17T02:02:12.416 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.414+0000 7fd54976b380 0 bucket checkpoint complete 2021-09-17T02:02:12.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.430 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:12.481 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:12.481 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:12.499 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.499+0000 7f4c10ee9380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:12.499 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.499+0000 7f4c10ee9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:12.499 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.499+0000 7f4c10ee9380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:12.500 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.499+0000 7f4c10ee9380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:12.544 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.544+0000 7f4c10ee9380 1 bucket sync caught up with source: 2021-09-17T02:02:12.545 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000003.66.9, , 00000000010.233.5, , 00000000009.594.9, , , , , ] 2021-09-17T02:02:12.545 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000003.66.9, , 00000000010.233.5, , 00000000009.594.9, , , , , ] 2021-09-17T02:02:12.545 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.544+0000 7f4c10ee9380 0 bucket checkpoint complete 2021-09-17T02:02:12.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.559 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:12.609 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:12.609 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:12.627 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.627+0000 7f792d710380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:12.627 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.627+0000 7f792d710380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:12.627 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.627+0000 7f792d710380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:12.628 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.627+0000 7f792d710380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:12.653 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.652+0000 7f792d710380 1 bucket sync caught up with source: 2021-09-17T02:02:12.653 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000003.66.9, , 00000000010.233.5, , 00000000009.594.9, , , , , ] 2021-09-17T02:02:12.653 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000003.66.9, , 00000000010.233.5, , 00000000009.594.9, , , , , ] 2021-09-17T02:02:12.653 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.652+0000 7f792d710380 0 bucket checkpoint complete 2021-09-17T02:02:12.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.666 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:12.716 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:12.716 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:12.735 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.734+0000 7effe8813380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:12.735 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.734+0000 7effe8813380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:12.735 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.734+0000 7effe8813380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:12.735 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.734+0000 7effe8813380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:12.759 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.759+0000 7effe8813380 1 bucket sync caught up with source: 2021-09-17T02:02:12.759 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000003.84.9, , 00000000003.219.9, , 00000000014.482.5, , , , , ] 2021-09-17T02:02:12.760 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000003.84.9, , 00000000003.219.9, , 00000000014.482.5, , , , , ] 2021-09-17T02:02:12.760 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:12.759+0000 7effe8813380 0 bucket checkpoint complete 2021-09-17T02:02:12.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:12.773 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:12.827 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:12.828 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.847+0000 7f3270b2f380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:12.848 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.847+0000 7f3270b2f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.847+0000 7f3270b2f380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:12.849 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.847+0000 7f3270b2f380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:12.874 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.873+0000 7f3270b2f380 1 bucket sync caught up with source: 2021-09-17T02:02:12.874 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000003.84.9, , 00000000003.219.9, , 00000000014.482.5, , , , , ] 2021-09-17T02:02:12.874 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000003.84.9, , 00000000003.219.9, , 00000000014.482.5, , , , , ] 2021-09-17T02:02:12.875 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:12.873+0000 7f3270b2f380 0 bucket checkpoint complete 2021-09-17T02:02:12.887 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-17 zones={test-zone2, test-zone1} 2021-09-17T02:02:12.891 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:12.894 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:12.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:12.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:12.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:12.896 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:12.899 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:12.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:12.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:12.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:12.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:12.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:12.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:13.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:13.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:13.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:13.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:13.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:13.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:13.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:13.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:13.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:13.778 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-17 zones={test-zone2, test-zone1} 2021-09-17T02:02:13.778 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-17 zones={test-zone3, test-zone1} 2021-09-17T02:02:13.784 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:13.835 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:13.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:13.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:13.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:13.837 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:13.841 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:13.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:13.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:13.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:13.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:13.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:13.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:14.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:14.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:14.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:14.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:14.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:14.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:14.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:14.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:14.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:14.782 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-17 zones={test-zone3, test-zone1} 2021-09-17T02:02:14.783 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-17 zones={test-zone3, test-zone2} 2021-09-17T02:02:14.788 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:14.791 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:14.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:14.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:14.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:14.793 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:14.839 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:14.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:14.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:14.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:14.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:14.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:14.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:15.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:15.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:15.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:15.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:15.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:15.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:15.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:15.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:15.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:15.782 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-17 zones={test-zone3, test-zone2} 2021-09-17T02:02:15.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:15.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:15.783 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:15.840 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:15.861 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:15.861+0000 7f8fd4ae5380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:15.862 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:15.861+0000 7f8fd4ae5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:15.862 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:15.861+0000 7f8fd4ae5380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:15.862 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:15.861+0000 7f8fd4ae5380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:15.888+0000 7f8fd4ae5380 1 bucket sync caught up with source: 2021-09-17T02:02:15.888 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000010.24.5, , 00000000003.10.9, , 00000000007.15.9, , , , , ] 2021-09-17T02:02:15.889 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000010.24.5, , 00000000003.10.9, , 00000000007.15.9, , , , , ] 2021-09-17T02:02:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:15.888+0000 7f8fd4ae5380 0 bucket checkpoint complete 2021-09-17T02:02:15.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:15.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:15.903 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:15.959 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:15.960 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:15.981 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:15.981+0000 7f5ba1398380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:15.981 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:15.981+0000 7f5ba1398380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:15.982 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:15.981+0000 7f5ba1398380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:15.982 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:15.981+0000 7f5ba1398380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:16.007 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.007+0000 7f5ba1398380 1 bucket sync caught up with source: 2021-09-17T02:02:16.007 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000010.24.5, , 00000000003.10.9, , 00000000007.15.9, , , , , ] 2021-09-17T02:02:16.008 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000010.24.5, , 00000000003.10.9, , 00000000007.15.9, , , , , ] 2021-09-17T02:02:16.008 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.007+0000 7f5ba1398380 0 bucket checkpoint complete 2021-09-17T02:02:16.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:16.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:16.022 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:16.076 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:16.076 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:16.096 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.095+0000 7f9d5416e380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:16.096 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.095+0000 7f9d5416e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:16.096 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.095+0000 7f9d5416e380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:02:16.097 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.095+0000 7f9d5416e380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:16.120 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.120+0000 7f9d5416e380 1 bucket sync caught up with source: 2021-09-17T02:02:16.121 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000010.24.5, , 00000000003.10.9, , 00000000007.15.9, , , , , ] 2021-09-17T02:02:16.121 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000010.24.5, , 00000000003.10.9, , 00000000007.15.9, , , , , ] 2021-09-17T02:02:16.121 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.120+0000 7f9d5416e380 0 bucket checkpoint complete 2021-09-17T02:02:16.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:16.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:16.133 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:16.185 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:16.185 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:16.204 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.203+0000 7f3050633380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:16.204 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.203+0000 7f3050633380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:16.204 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.203+0000 7f3050633380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:16.204 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.203+0000 7f3050633380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:16.228 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.227+0000 7f3050633380 1 bucket sync caught up with source: 2021-09-17T02:02:16.228 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000003.136.9, , 00000000010.150.5, , 00000000010.81.5, , , , , ] 2021-09-17T02:02:16.228 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000003.136.9, , 00000000010.150.5, , 00000000010.81.5, , , , , ] 2021-09-17T02:02:16.229 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.227+0000 7f3050633380 0 bucket checkpoint complete 2021-09-17T02:02:16.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:16.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:16.242 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:16.295 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:16.295 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:16.314 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.314+0000 7f7bae3d0380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:16.315 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.314+0000 7f7bae3d0380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:16.315 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.314+0000 7f7bae3d0380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:16.315 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.314+0000 7f7bae3d0380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:16.339 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.339+0000 7f7bae3d0380 1 bucket sync caught up with source: 2021-09-17T02:02:16.340 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000003.136.9, , 00000000010.150.5, , 00000000010.81.5, , , , , ] 2021-09-17T02:02:16.340 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000003.136.9, , 00000000010.150.5, , 00000000010.81.5, , , , , ] 2021-09-17T02:02:16.340 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.339+0000 7f7bae3d0380 0 bucket checkpoint complete 2021-09-17T02:02:16.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:16.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:16.353 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:16.403 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:16.403 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:16.421 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.421+0000 7fc17c5f5380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:16.422 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.421+0000 7fc17c5f5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:16.422 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.421+0000 7fc17c5f5380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:16.422 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.421+0000 7fc17c5f5380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:16.446 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.446+0000 7fc17c5f5380 1 bucket sync caught up with source: 2021-09-17T02:02:16.446 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000003.117.9, , 00000000003.166.9, , 00000000016.71.5, , , , , ] 2021-09-17T02:02:16.446 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000003.117.9, , 00000000003.166.9, , 00000000016.71.5, , , , , ] 2021-09-17T02:02:16.447 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:16.446+0000 7fc17c5f5380 0 bucket checkpoint complete 2021-09-17T02:02:16.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:16.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:16.460 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:16.513 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:16.513 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:16.532 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.531+0000 7f60c1faa380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:16.532 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.531+0000 7f60c1faa380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:16.533 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.531+0000 7f60c1faa380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:16.533 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.531+0000 7f60c1faa380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:16.558 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.557+0000 7f60c1faa380 1 bucket sync caught up with source: 2021-09-17T02:02:16.559 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000003.117.9, , 00000000003.166.9, , 00000000016.71.5, , , , , ] 2021-09-17T02:02:16.559 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000003.117.9, , 00000000003.166.9, , 00000000016.71.5, , , , , ] 2021-09-17T02:02:16.559 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:16.558+0000 7f60c1faa380 0 bucket checkpoint complete 2021-09-17T02:02:16.571 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-18 zones={test-zone2, test-zone1} 2021-09-17T02:02:16.576 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:16.579 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:16.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:16.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:16.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:16.580 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:16.583 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:16.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:16.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:16.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:16.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:16.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:16.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:16.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:16.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:16.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:16.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:17.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:17.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:17.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:17.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:17.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:17.470 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-18 zones={test-zone2, test-zone1} 2021-09-17T02:02:17.470 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-18 zones={test-zone3, test-zone1} 2021-09-17T02:02:17.476 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:17.480 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:17.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:17.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:17.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:17.482 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:17.486 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:17.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:17.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:17.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:17.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:17.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:17.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:17.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:17.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:17.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:17.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:17.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:18.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:18.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:18.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:18.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:18.354 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-18 zones={test-zone3, test-zone1} 2021-09-17T02:02:18.355 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-18 zones={test-zone3, test-zone2} 2021-09-17T02:02:18.360 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:18.363 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:18.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:18.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:18.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:18.365 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:18.411 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:18.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:18.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:18.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:18.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:18.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:18.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:18.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:18.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:18.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:18.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:18.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:18.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:19.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:19.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:19.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:19.342 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-18 zones={test-zone3, test-zone2} 2021-09-17T02:02:20.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:20.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:20.007 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:20.071 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:20.071 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:20.090 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:20.090+0000 7fb91a2a6380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:20.090+0000 7fb91a2a6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:20.090+0000 7fb91a2a6380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:20.091 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:20.090+0000 7fb91a2a6380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:20.968 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:02:20.967+0000 7f62b31ff700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-17T02:02:21.120 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.120+0000 7fb91a2a6380 1 bucket sync caught up with source: 2021-09-17T02:02:21.121 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000011.33.5, , 00000000007.25.5, , 00000000013.35.5, , , , , ] 2021-09-17T02:02:21.121 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000011.33.5, , 00000000007.25.5, , 00000000013.35.5, , , , , ] 2021-09-17T02:02:21.121 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.120+0000 7fb91a2a6380 0 bucket checkpoint complete 2021-09-17T02:02:21.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.135 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:21.192 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:21.193 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:21.214 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.214+0000 7f434904f380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:21.215 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.214+0000 7f434904f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:21.215 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.214+0000 7f434904f380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:21.215 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.214+0000 7f434904f380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:21.241 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.241+0000 7f434904f380 1 bucket sync caught up with source: 2021-09-17T02:02:21.242 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000011.33.5, , 00000000007.25.5, , 00000000013.35.5, , , , , ] 2021-09-17T02:02:21.242 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000011.33.5, , 00000000007.25.5, , 00000000013.35.5, , , , , ] 2021-09-17T02:02:21.243 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.241+0000 7f434904f380 0 bucket checkpoint complete 2021-09-17T02:02:21.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.256 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:21.307 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:21.308 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.326+0000 7f3e371b0380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.326+0000 7f3e371b0380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.326+0000 7f3e371b0380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:02:21.327 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.326+0000 7f3e371b0380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:21.351 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.350+0000 7f3e371b0380 1 bucket sync caught up with source: 2021-09-17T02:02:21.351 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000011.33.5, , 00000000007.25.5, , 00000000013.35.5, , , , , ] 2021-09-17T02:02:21.352 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000011.33.5, , 00000000007.25.5, , 00000000013.35.5, , , , , ] 2021-09-17T02:02:21.352 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.350+0000 7f3e371b0380 0 bucket checkpoint complete 2021-09-17T02:02:21.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.364 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:21.416 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:21.417 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:21.435 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.435+0000 7f840a151380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:21.436 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.435+0000 7f840a151380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:21.436 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.435+0000 7f840a151380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:21.436 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.435+0000 7f840a151380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:21.461 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.461+0000 7f840a151380 1 bucket sync caught up with source: 2021-09-17T02:02:21.462 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000004.221.5, , 00000000011.288.5, , 00000000011.243.5, , , , , ] 2021-09-17T02:02:21.462 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000004.221.5, , 00000000011.288.5, , 00000000011.243.5, , , , , ] 2021-09-17T02:02:21.462 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.461+0000 7f840a151380 0 bucket checkpoint complete 2021-09-17T02:02:21.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.475 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:21.525 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:21.525 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:21.543 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.543+0000 7fca28081380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:21.544 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.543+0000 7fca28081380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:21.544 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.543+0000 7fca28081380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:21.544 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.543+0000 7fca28081380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:21.568 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.567+0000 7fca28081380 1 bucket sync caught up with source: 2021-09-17T02:02:21.568 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000004.221.5, , 00000000011.288.5, , 00000000011.243.5, , , , , ] 2021-09-17T02:02:21.569 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000004.221.5, , 00000000011.288.5, , 00000000011.243.5, , , , , ] 2021-09-17T02:02:21.569 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.567+0000 7fca28081380 0 bucket checkpoint complete 2021-09-17T02:02:21.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.581 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:21.631 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:21.631 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:21.649 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.649+0000 7fa4b231b380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:21.650 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.649+0000 7fa4b231b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:21.650 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.649+0000 7fa4b231b380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:21.650 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.649+0000 7fa4b231b380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:21.674 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.674+0000 7fa4b231b380 1 bucket sync caught up with source: 2021-09-17T02:02:21.675 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000004.239.5, , 00000000007.217.5, , 00000000015.223.5, , , , , ] 2021-09-17T02:02:21.675 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000004.239.5, , 00000000007.217.5, , 00000000015.223.5, , , , , ] 2021-09-17T02:02:21.675 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:21.674+0000 7fa4b231b380 0 bucket checkpoint complete 2021-09-17T02:02:21.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:21.688 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:21.739 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:21.740 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:21.758 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.757+0000 7f1c8647e380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:21.758 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.757+0000 7f1c8647e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:21.759 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.757+0000 7f1c8647e380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:21.759 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.757+0000 7f1c8647e380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:21.782 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.781+0000 7f1c8647e380 1 bucket sync caught up with source: 2021-09-17T02:02:21.782 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000004.239.5, , 00000000007.217.5, , 00000000015.223.5, , , , , ] 2021-09-17T02:02:21.782 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000004.239.5, , 00000000007.217.5, , 00000000015.223.5, , , , , ] 2021-09-17T02:02:21.783 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:21.781+0000 7f1c8647e380 0 bucket checkpoint complete 2021-09-17T02:02:21.794 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-16 zones={test-zone2, test-zone1} 2021-09-17T02:02:21.799 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:21.802 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:21.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:21.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:21.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:21.804 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:21.851 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:21.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:21.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:21.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:21.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:21.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:21.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:21.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:22.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:22.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:22.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:22.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:22.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:22.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:22.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:22.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:22.726 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-16 zones={test-zone2, test-zone1} 2021-09-17T02:02:22.727 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-16 zones={test-zone3, test-zone1} 2021-09-17T02:02:22.731 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:22.735 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:22.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:22.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:22.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:22.737 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:22.740 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:22.740 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:22.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:22.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:22.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:22.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:22.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:22.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:22.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:23.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:23.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:23.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:23.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:23.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:23.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:23.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:23.615 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-16 zones={test-zone3, test-zone1} 2021-09-17T02:02:23.615 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-16 zones={test-zone3, test-zone2} 2021-09-17T02:02:23.621 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:23.667 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:23.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:23.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:23.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:23.669 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:23.673 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:23.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:23.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:23.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:23.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:23.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:23.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:23.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:23.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:23.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:24.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:24.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:24.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:24.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:24.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:24.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:24.566 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-16 zones={test-zone3, test-zone2} 2021-09-17T02:02:24.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:24.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:24.568 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:24.625 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:24.647 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.646+0000 7f8d4e344380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:24.647 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.646+0000 7f8d4e344380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:24.648 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.646+0000 7f8d4e344380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:24.648 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.646+0000 7f8d4e344380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:24.673 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.673+0000 7f8d4e344380 1 bucket sync caught up with source: 2021-09-17T02:02:24.674 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000011.28.5, , 00000000004.16.5, , 00000000008.29.5, , , , , ] 2021-09-17T02:02:24.674 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000011.28.5, , 00000000004.16.5, , 00000000008.29.5, , , , , ] 2021-09-17T02:02:24.674 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.673+0000 7f8d4e344380 0 bucket checkpoint complete 2021-09-17T02:02:24.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:24.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:24.688 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:24.745 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:24.746 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:24.767 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:24.765+0000 7f2e4cdcf380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:24.767 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:24.765+0000 7f2e4cdcf380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:24.768 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:24.765+0000 7f2e4cdcf380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:24.768 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:24.765+0000 7f2e4cdcf380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:24.793 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:24.792+0000 7f2e4cdcf380 1 bucket sync caught up with source: 2021-09-17T02:02:24.793 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000011.28.5, , 00000000004.16.5, , 00000000008.29.5, , , , , ] 2021-09-17T02:02:24.794 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000011.28.5, , 00000000004.16.5, , 00000000008.29.5, , , , , ] 2021-09-17T02:02:24.794 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:24.792+0000 7f2e4cdcf380 0 bucket checkpoint complete 2021-09-17T02:02:24.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:24.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:24.808 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:24.860 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:24.861 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:24.879 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.878+0000 7fca294e2380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:24.880 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.878+0000 7fca294e2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:24.880 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.878+0000 7fca294e2380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:02:24.881 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.878+0000 7fca294e2380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:24.904 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.903+0000 7fca294e2380 1 bucket sync caught up with source: 2021-09-17T02:02:24.904 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000011.28.5, , 00000000004.16.5, , 00000000008.29.5, , , , , ] 2021-09-17T02:02:24.905 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000011.28.5, , 00000000004.16.5, , 00000000008.29.5, , , , , ] 2021-09-17T02:02:24.905 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:24.903+0000 7fca294e2380 0 bucket checkpoint complete 2021-09-17T02:02:24.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:24.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:24.918 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:24.976 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:24.976 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:24.996 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:24.994+0000 7f70b32cc380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:24.996 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:24.994+0000 7f70b32cc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:24.996 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:24.994+0000 7f70b32cc380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:24.996 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:24.994+0000 7f70b32cc380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:25.021 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.020+0000 7f70b32cc380 1 bucket sync caught up with source: 2021-09-17T02:02:25.021 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000004.69.5, , 00000000011.240.5, , 00000000010.597.5, , , , , ] 2021-09-17T02:02:25.022 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000004.69.5, , 00000000011.240.5, , 00000000010.597.5, , , , , ] 2021-09-17T02:02:25.022 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.020+0000 7f70b32cc380 0 bucket checkpoint complete 2021-09-17T02:02:25.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:25.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:25.035 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:25.085 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:25.086 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:25.104 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.102+0000 7f83a5282380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:25.104 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.102+0000 7f83a5282380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:25.104 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.102+0000 7f83a5282380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:25.105 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.102+0000 7f83a5282380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:25.128 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.127+0000 7f83a5282380 1 bucket sync caught up with source: 2021-09-17T02:02:25.129 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000004.69.5, , 00000000011.240.5, , 00000000010.597.5, , , , , ] 2021-09-17T02:02:25.129 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000004.69.5, , 00000000011.240.5, , 00000000010.597.5, , , , , ] 2021-09-17T02:02:25.129 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.127+0000 7f83a5282380 0 bucket checkpoint complete 2021-09-17T02:02:25.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:25.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:25.143 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:25.193 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:25.193 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:25.211 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.210+0000 7f19a1212380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:25.211 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.210+0000 7f19a1212380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:25.212 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.210+0000 7f19a1212380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:25.212 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.210+0000 7f19a1212380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:25.237 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.235+0000 7f19a1212380 1 bucket sync caught up with source: 2021-09-17T02:02:25.237 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000004.87.5, , 00000000004.222.5, , 00000000015.485.5, , , , , ] 2021-09-17T02:02:25.237 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000004.87.5, , 00000000004.222.5, , 00000000015.485.5, , , , , ] 2021-09-17T02:02:25.238 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:25.235+0000 7f19a1212380 0 bucket checkpoint complete 2021-09-17T02:02:25.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:25.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:25.251 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:25.303 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:25.321 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:25.321+0000 7f5cbd6a8380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:25.322 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:25.321+0000 7f5cbd6a8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:25.322 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:25.321+0000 7f5cbd6a8380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:25.322 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:25.321+0000 7f5cbd6a8380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:25.345 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:25.345+0000 7f5cbd6a8380 1 bucket sync caught up with source: 2021-09-17T02:02:25.346 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000004.87.5, , 00000000004.222.5, , 00000000015.485.5, , , , , ] 2021-09-17T02:02:25.346 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000004.87.5, , 00000000004.222.5, , 00000000015.485.5, , , , , ] 2021-09-17T02:02:25.346 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:25.345+0000 7f5cbd6a8380 0 bucket checkpoint complete 2021-09-17T02:02:25.358 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-17 zones={test-zone2, test-zone1} 2021-09-17T02:02:25.363 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:25.366 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:25.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:25.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:25.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:25.368 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:25.371 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:25.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:25.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:25.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:25.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:25.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:25.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:25.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:25.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:25.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:25.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:25.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:25.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:26.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:26.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:26.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:26.258 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-17 zones={test-zone2, test-zone1} 2021-09-17T02:02:26.259 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-17 zones={test-zone3, test-zone1} 2021-09-17T02:02:26.264 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:26.311 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:26.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:26.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:26.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:26.312 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:26.316 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:26.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:26.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:26.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:26.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:26.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:26.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:26.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:26.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:26.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:26.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:26.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:26.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:26.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:27.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:27.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:27.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-17 zones={test-zone3, test-zone1} 2021-09-17T02:02:27.243 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-17 zones={test-zone3, test-zone2} 2021-09-17T02:02:27.248 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:27.295 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:27.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:27.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:27.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:27.297 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:27.300 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:27.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:27.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:27.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:27.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:27.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:27.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:27.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:27.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:27.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:27.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:27.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:27.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:27.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:28.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:28.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:28.238 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-17 zones={test-zone3, test-zone2} 2021-09-17T02:02:28.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.240 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:28.297 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:28.298 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:28.318 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.317+0000 7f38849a6380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:28.319 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.317+0000 7f38849a6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:28.319 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.317+0000 7f38849a6380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:28.319 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.317+0000 7f38849a6380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:28.346 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.344+0000 7f38849a6380 1 bucket sync caught up with source: 2021-09-17T02:02:28.346 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000011.31.5, , 00000000004.13.5, , 00000000008.18.5, , , , , ] 2021-09-17T02:02:28.347 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000011.31.5, , 00000000004.13.5, , 00000000008.18.5, , , , , ] 2021-09-17T02:02:28.347 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.344+0000 7f38849a6380 0 bucket checkpoint complete 2021-09-17T02:02:28.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.360 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:28.418 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:28.419 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:28.440 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.439+0000 7fa01bf1e380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:28.440 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.439+0000 7fa01bf1e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:28.441 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.439+0000 7fa01bf1e380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:28.441 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.439+0000 7fa01bf1e380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:28.467 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.466+0000 7fa01bf1e380 1 bucket sync caught up with source: 2021-09-17T02:02:28.467 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000011.31.5, , 00000000004.13.5, , 00000000008.18.5, , , , , ] 2021-09-17T02:02:28.468 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000011.31.5, , 00000000004.13.5, , 00000000008.18.5, , , , , ] 2021-09-17T02:02:28.468 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.466+0000 7fa01bf1e380 0 bucket checkpoint complete 2021-09-17T02:02:28.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.482 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:28.536 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.554+0000 7ff6aa0d7380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.554+0000 7ff6aa0d7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.554+0000 7ff6aa0d7380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:02:28.556 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.554+0000 7ff6aa0d7380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:28.580 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.578+0000 7ff6aa0d7380 1 bucket sync caught up with source: 2021-09-17T02:02:28.580 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000011.31.5, , 00000000004.13.5, , 00000000008.18.5, , , , , ] 2021-09-17T02:02:28.581 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000011.31.5, , 00000000004.13.5, , 00000000008.18.5, , , , , ] 2021-09-17T02:02:28.581 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.578+0000 7ff6aa0d7380 0 bucket checkpoint complete 2021-09-17T02:02:28.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.593 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:28.646 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:28.646 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:28.665 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.664+0000 7f57592d5380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:28.665 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.664+0000 7f57592d5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:28.665 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.664+0000 7f57592d5380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:28.666 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.664+0000 7f57592d5380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:28.691 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.689+0000 7f57592d5380 1 bucket sync caught up with source: 2021-09-17T02:02:28.691 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000004.139.5, , 00000000011.157.5, , 00000000011.84.5, , , , , ] 2021-09-17T02:02:28.691 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000004.139.5, , 00000000011.157.5, , 00000000011.84.5, , , , , ] 2021-09-17T02:02:28.692 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.689+0000 7f57592d5380 0 bucket checkpoint complete 2021-09-17T02:02:28.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.705 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:28.755 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:28.756 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:28.774 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.772+0000 7ff2e4764380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:28.774 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.772+0000 7ff2e4764380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:28.775 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.772+0000 7ff2e4764380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:28.775 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.772+0000 7ff2e4764380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:28.798 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.797+0000 7ff2e4764380 1 bucket sync caught up with source: 2021-09-17T02:02:28.798 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000004.139.5, , 00000000011.157.5, , 00000000011.84.5, , , , , ] 2021-09-17T02:02:28.799 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000004.139.5, , 00000000011.157.5, , 00000000011.84.5, , , , , ] 2021-09-17T02:02:28.799 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.797+0000 7ff2e4764380 0 bucket checkpoint complete 2021-09-17T02:02:28.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.812 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:28.861 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:28.862 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:28.879 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.878+0000 7f855773d380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:28.880 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.878+0000 7f855773d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:28.880 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.878+0000 7f855773d380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:28.880 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.878+0000 7f855773d380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:28.904 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.903+0000 7f855773d380 1 bucket sync caught up with source: 2021-09-17T02:02:28.904 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, 00000000004.120.5, , 00000000004.169.5, , 00000000017.74.5, , , , , ] 2021-09-17T02:02:28.904 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, 00000000004.120.5, , 00000000004.169.5, , 00000000017.74.5, , , , , ] 2021-09-17T02:02:28.905 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:28.903+0000 7f855773d380 0 bucket checkpoint complete 2021-09-17T02:02:28.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:28.918 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:28.971 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:28.989 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.988+0000 7fe64c262380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:28.990 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.988+0000 7fe64c262380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:28.990 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.988+0000 7fe64c262380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:28.990 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:28.988+0000 7fe64c262380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:29.014 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:29.013+0000 7fe64c262380 1 bucket sync caught up with source: 2021-09-17T02:02:29.015 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, 00000000004.120.5, , 00000000004.169.5, , 00000000017.74.5, , , , , ] 2021-09-17T02:02:29.015 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, 00000000004.120.5, , 00000000004.169.5, , 00000000017.74.5, , , , , ] 2021-09-17T02:02:29.015 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:29.013+0000 7fe64c262380 0 bucket checkpoint complete 2021-09-17T02:02:29.027 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-18 zones={test-zone2, test-zone1} 2021-09-17T02:02:29.032 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:29.035 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:29.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:29.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:29.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:29.037 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:29.083 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:29.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:29.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:29.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:29.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:29.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:29.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:29.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:29.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:29.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:29.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:29.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:29.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:29.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:29.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:29.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:29.962 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-18 zones={test-zone2, test-zone1} 2021-09-17T02:02:29.963 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-18 zones={test-zone3, test-zone1} 2021-09-17T02:02:29.968 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:29.972 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:29.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:29.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:29.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:29.974 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:29.977 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:29.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:29.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:29.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:30.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:30.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:30.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:30.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:30.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:30.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:30.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:30.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:30.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:30.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:30.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:30.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:30.894 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-18 zones={test-zone3, test-zone1} 2021-09-17T02:02:30.894 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-18 zones={test-zone3, test-zone2} 2021-09-17T02:02:30.900 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:30.903 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:30.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:30.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:30.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:30.905 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:30.951 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:30.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T02:02:30.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T02:02:30.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T02:02:30.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:30.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:31.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:31.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:31.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:31.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T02:02:31.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:31.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:31.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T02:02:31.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:31.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:31.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T02:02:31.878 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-18 zones={test-zone3, test-zone2} 2021-09-17T02:02:31.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-09-17T02:02:31.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:02:31.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:02:31.903 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:02:31.950 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:31.950 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:31.990 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:02:32.002 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:02:32.003 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:02:32.003 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:02:32.003 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:02:32.003 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:02:32.004 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:02:32.004 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844151.901011_172.1", 2021-09-17T02:02:32.004 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:02:31.901011Z" 2021-09-17T02:02:32.004 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:02:32.004 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:02:32.005 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:02:32.005 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:02:32.005 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:02:32.005 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:02:32.006 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:02:32.006 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:02:32.006 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:02:32.006 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:02:32.007 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:02:32.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:02:32.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:02:32.009 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:02:32.059 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:32.060 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:32.116 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:02:32.116 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:02:32.117 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:02:32.117 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:02:32.117 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:02:32.117 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:02:32.118 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:32.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:32.118 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:02:32.118 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:32.119 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:02:32.119 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:32.119 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:32.119 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:02:32.120 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:32.120 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:02:32.120 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:32.120 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:02:32.120 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:32.121 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:32.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:32.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:32.121 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:02:32.122 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:32.122 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:32.122 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844151.886115_167.1", 2021-09-17T02:02:32.122 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:32.123 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:02:32.123 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:32.123 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:31.886115Z", 2021-09-17T02:02:32.123 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:32.123 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:32.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:32.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:32.124 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:02:32.124 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:32.125 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:32.125 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:02:32.125 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:32.125 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:02:32.125 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:32.126 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:02:32.126 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:32.126 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:32.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:32.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:32.127 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:02:32.127 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:32.127 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:32.127 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:02:32.128 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:32.128 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:02:32.128 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:32.128 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:02:32.128 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:32.129 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:32.129 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:32.129 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:02:32.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:32.129 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:02:32.130 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:02:32.130 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:02:32.130 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:32.130 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:02:32.131 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844151.901011_172.1 target=1_1631844151.886115_167.1 2021-09-17T02:02:37.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:02:37.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:02:37.134 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:02:37.185 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:37.186 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:37.240 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:02:37.241 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:02:37.241 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:02:37.241 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:02:37.242 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:02:37.242 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:02:37.242 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:37.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:37.243 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:02:37.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:37.243 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:02:37.243 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:37.244 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:37.244 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:02:37.244 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:37.244 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:02:37.244 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:37.245 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:02:37.245 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:37.245 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:37.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:37.246 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:02:37.246 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:37.246 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:37.246 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844151.901011_172.1", 2021-09-17T02:02:37.247 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:37.247 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:02:37.247 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:37.247 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:31.901011Z", 2021-09-17T02:02:37.248 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:37.248 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:37.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:37.248 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:02:37.249 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:37.249 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:37.249 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:02:37.249 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:37.250 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:02:37.250 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:37.250 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:02:37.250 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:37.250 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:37.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:37.251 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:02:37.251 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:37.252 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:37.252 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:02:37.252 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:37.252 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:02:37.252 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:37.253 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:02:37.253 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:37.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.254 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:02:37.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:37.254 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:02:37.254 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:02:37.254 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:02:37.255 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.255 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:02:37.256 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:02:37.257 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:02:37.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:02:37.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:02:37.258 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:02:37.315 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:37.315 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:37.369 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:02:37.370 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:02:37.370 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:02:37.370 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:02:37.371 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:02:37.371 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:02:37.371 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:02:37.371 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:02:37.372 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:02:37.372 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:02:37.372 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:02:37.372 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:02:37.373 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:02:37.373 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:02:37.373 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:02:37.373 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:02:37.373 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:02:37.374 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:02:37.374 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:02:37.374 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:02:37.374 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:02:37.375 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:02:37.375 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:02:37.375 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:02:37.375 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:02:37.376 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844151.901011_172.1", 2021-09-17T02:02:37.376 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:02:37.376 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:02:37.376 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:02:37.376 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:02:31.901011Z", 2021-09-17T02:02:37.377 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:02:37.377 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:02:37.377 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:02:37.377 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:02:37.378 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:02:37.378 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:02:37.378 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:02:37.378 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:02:37.378 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:02:37.379 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:02:37.379 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:02:37.379 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:02:37.379 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:02:37.380 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:02:37.380 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:02:37.380 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:02:37.380 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:02:37.380 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:02:37.381 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:02:37.381 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:02:37.381 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:02:37.381 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:02:37.382 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:02:37.382 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:02:37.382 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:02:37.382 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:02:37.383 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:02:37.383 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:02:37.383 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:02:37.383 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:02:37.383 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:02:37.384 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:02:37.384 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:02:37.384 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:02:37.385 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:02:37.386 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:02:37.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:02:37.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:02:37.387 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:02:37.436 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:37.436 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:37.487 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:02:37.487 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:02:37.487 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:02:37.488 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:02:37.488 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:02:37.488 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:02:37.488 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:37.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:37.489 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:02:37.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:37.489 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:02:37.489 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:37.490 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:37.490 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:02:37.490 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:37.490 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:02:37.491 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:37.491 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:02:37.491 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:37.491 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:37.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:37.492 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:02:37.492 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:37.492 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:37.493 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844151.901011_172.1", 2021-09-17T02:02:37.493 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:37.493 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:02:37.493 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:37.494 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:31.901011Z", 2021-09-17T02:02:37.494 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:37.494 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:37.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:37.495 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:02:37.495 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:37.495 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:37.495 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:02:37.496 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:37.496 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:02:37.496 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:37.496 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:02:37.496 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:37.497 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:37.497 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:02:37.497 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:02:37.498 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:02:37.498 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:02:37.498 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:02:37.498 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:02:37.498 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:02:37.499 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:02:37.499 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:02:37.499 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:02:37.499 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.500 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.500 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:02:37.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:02:37.500 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:02:37.501 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:02:37.501 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:02:37.501 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:02:37.501 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:02:37.503 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:02:37.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:37.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:37.556 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:37.610 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:37.610 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:37.629 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:37.627+0000 7f3287ee3380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:37.627+0000 7f3287ee3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:37.627+0000 7f3287ee3380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:37.630 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:37.627+0000 7f3287ee3380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:37.654 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:37.652+0000 7f3287ee3380 1 waiting for incremental sync to catch up: 2021-09-17T02:02:37.654 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , , , ] 2021-09-17T02:02:37.655 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , 00000000002.8.10, , , , , ] 2021-09-17T02:02:42.655 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:42.654+0000 7f3287ee3380 1 bucket sync caught up with source: 2021-09-17T02:02:42.656 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , 00000000006.16.9, , , , , ] 2021-09-17T02:02:42.656 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , 00000000002.8.10, , , , , ] 2021-09-17T02:02:42.656 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:42.654+0000 7f3287ee3380 0 bucket checkpoint complete 2021-09-17T02:02:42.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:42.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:42.673 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:42.732 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:42.732 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:42.753 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:42.751+0000 7f738c68c380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:42.753 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:42.751+0000 7f738c68c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:42.754 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:42.751+0000 7f738c68c380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:42.754 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:42.751+0000 7f738c68c380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:42.779 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:42.778+0000 7f738c68c380 1 bucket sync caught up with source: 2021-09-17T02:02:42.779 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000006.16.9, , , , , ] 2021-09-17T02:02:42.780 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2021-09-17T02:02:42.780 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:42.778+0000 7f738c68c380 0 bucket checkpoint complete 2021-09-17T02:02:42.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:42.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:42.794 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:42.851 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:42.852 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:42.873 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:42.871+0000 7ff1a3d9c380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:42.873 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:42.871+0000 7ff1a3d9c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:42.873 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:42.871+0000 7ff1a3d9c380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:02:42.874 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:42.871+0000 7ff1a3d9c380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:42.900 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:42.898+0000 7ff1a3d9c380 1 bucket sync caught up with source: 2021-09-17T02:02:42.900 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , 00000000006.16.9, , , , , ] 2021-09-17T02:02:42.901 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2021-09-17T02:02:42.901 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:42.898+0000 7ff1a3d9c380 0 bucket checkpoint complete 2021-09-17T02:02:42.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:42.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:42.914 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:42.969 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:42.969 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:42.988 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:42.986+0000 7fd4cf4bb380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:42.988 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:42.986+0000 7fd4cf4bb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:42.988 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:42.986+0000 7fd4cf4bb380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:42.988 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:42.986+0000 7fd4cf4bb380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:43.014 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.012+0000 7fd4cf4bb380 1 bucket sync caught up with source: 2021-09-17T02:02:43.014 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000004.258.9, , , , , ] 2021-09-17T02:02:43.014 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000004.258.9, , , , , ] 2021-09-17T02:02:43.014 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.012+0000 7fd4cf4bb380 0 bucket checkpoint complete 2021-09-17T02:02:43.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:43.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:43.027 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:43.078 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:43.078 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:43.096 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.095+0000 7f7058480380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:43.096 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.095+0000 7f7058480380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:43.097 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.095+0000 7f7058480380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:43.097 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.095+0000 7f7058480380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:43.120 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.119+0000 7f7058480380 1 bucket sync caught up with source: 2021-09-17T02:02:43.121 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000004.258.9, , , , , ] 2021-09-17T02:02:43.121 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000004.258.9, , , , , ] 2021-09-17T02:02:43.121 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.119+0000 7f7058480380 0 bucket checkpoint complete 2021-09-17T02:02:43.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:43.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:43.134 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:43.184 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:43.184 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:43.202 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.201+0000 7f695addf380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:43.203 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.201+0000 7f695addf380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:43.203 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.201+0000 7f695addf380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:43.203 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.201+0000 7f695addf380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:43.227 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.225+0000 7f695addf380 1 bucket sync caught up with source: 2021-09-17T02:02:43.227 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000004.256.9, , , , , ] 2021-09-17T02:02:43.227 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000004.256.9, , , , , ] 2021-09-17T02:02:43.227 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:43.225+0000 7f695addf380 0 bucket checkpoint complete 2021-09-17T02:02:43.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:43.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:43.241 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:43.298 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:43.299 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:43.320 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:43.318+0000 7fef80fca380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:43.320 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:43.318+0000 7fef80fca380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:43.320 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:43.318+0000 7fef80fca380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:43.320 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:43.318+0000 7fef80fca380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:43.346 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:43.344+0000 7fef80fca380 1 bucket sync caught up with source: 2021-09-17T02:02:43.346 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , 00000000004.256.9, , , , , ] 2021-09-17T02:02:43.346 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , 00000000004.256.9, , , , , ] 2021-09-17T02:02:43.347 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:43.344+0000 7fef80fca380 0 bucket checkpoint complete 2021-09-17T02:02:43.359 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-19 zones={test-zone2, test-zone1} 2021-09-17T02:02:43.365 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:43.411 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:43.412 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:43.415 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:43.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:43.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:43.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:43.586 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-19 zones={test-zone2, test-zone1} 2021-09-17T02:02:43.587 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-19 zones={test-zone3, test-zone1} 2021-09-17T02:02:43.591 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:43.635 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:43.635 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:43.638 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:43.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:43.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:43.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:43.806 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-19 zones={test-zone3, test-zone1} 2021-09-17T02:02:43.806 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-19 zones={test-zone3, test-zone2} 2021-09-17T02:02:43.811 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:43.855 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:43.855 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:43.859 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:43.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:43.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:43.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:44.022 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-19 zones={test-zone3, test-zone2} 2021-09-17T02:02:45.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:45.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:45.401 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:45.453 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:45.453 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:45.472 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:45.470+0000 7fad98e46380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:45.472 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:45.470+0000 7fad98e46380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:45.473 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:45.470+0000 7fad98e46380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:45.473 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:45.470+0000 7fad98e46380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:45.502 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:45.501+0000 7fad98e46380 1 bucket sync caught up with source: 2021-09-17T02:02:45.503 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , 00000000006.16.9, , , 00000000055.142.11, , ] 2021-09-17T02:02:45.503 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , 00000000006.16.9, , , 00000000055.142.11, , ] 2021-09-17T02:02:45.503 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:45.501+0000 7fad98e46380 0 bucket checkpoint complete 2021-09-17T02:02:45.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:45.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:45.519 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:45.569 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:45.569 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:45.587 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:45.586+0000 7f5132e4f380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:45.588 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:45.586+0000 7f5132e4f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:45.588 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:45.586+0000 7f5132e4f380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:45.588 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:45.586+0000 7f5132e4f380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:45.612 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:45.610+0000 7f5132e4f380 1 waiting for incremental sync to catch up: 2021-09-17T02:02:45.612 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000006.16.9, , , 00000000055.142.11, , ] 2021-09-17T02:02:45.612 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000006.16.9, , , 00000000056.144.9, , ] 2021-09-17T02:02:50.613 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.612+0000 7f5132e4f380 1 bucket sync caught up with source: 2021-09-17T02:02:50.614 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000006.16.9, , , 00000000057.146.9, , ] 2021-09-17T02:02:50.614 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000006.16.9, , , 00000000056.144.9, , ] 2021-09-17T02:02:50.614 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.612+0000 7f5132e4f380 0 bucket checkpoint complete 2021-09-17T02:02:50.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:50.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:50.631 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:50.708 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:50.709 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:50.729 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:50.727+0000 7f14143da380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:50.727+0000 7f14143da380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:50.727+0000 7f14143da380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:02:50.730 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:50.727+0000 7f14143da380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:50.756 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:50.754+0000 7f14143da380 1 bucket sync caught up with source: 2021-09-17T02:02:50.756 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , 00000000006.16.9, , , 00000000057.146.9, , ] 2021-09-17T02:02:50.757 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , 00000000006.16.9, , , 00000000057.146.9, , ] 2021-09-17T02:02:50.757 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:50.754+0000 7f14143da380 0 bucket checkpoint complete 2021-09-17T02:02:50.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:50.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:50.771 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:50.830 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:50.831 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:50.852 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.850+0000 7fd91c90a380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:50.852 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.850+0000 7fd91c90a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:50.853 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.850+0000 7fd91c90a380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:50.853 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.850+0000 7fd91c90a380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:50.880 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.878+0000 7fd91c90a380 1 bucket sync caught up with source: 2021-09-17T02:02:50.880 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000004.258.9, , , 00000000054.482.9, , ] 2021-09-17T02:02:50.880 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000004.258.9, , , 00000000054.482.9, , ] 2021-09-17T02:02:50.881 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.878+0000 7fd91c90a380 0 bucket checkpoint complete 2021-09-17T02:02:50.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:50.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:50.895 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:50.949 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:50.949 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:50.968 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.967+0000 7f5a43387380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:50.969 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.967+0000 7f5a43387380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:50.969 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.967+0000 7f5a43387380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:02:50.969 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.967+0000 7f5a43387380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:50.993 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.992+0000 7f5a43387380 1 bucket sync caught up with source: 2021-09-17T02:02:50.994 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000004.258.9, , , 00000000054.482.9, , ] 2021-09-17T02:02:50.994 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000004.258.9, , , 00000000054.482.9, , ] 2021-09-17T02:02:50.994 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:50.992+0000 7f5a43387380 0 bucket checkpoint complete 2021-09-17T02:02:51.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:51.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:51.007 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:51.058 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:51.059 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:51.077 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:51.075+0000 7f8ab8f43380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:51.077 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:51.075+0000 7f8ab8f43380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:51.078 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:51.075+0000 7f8ab8f43380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:02:51.078 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:51.075+0000 7f8ab8f43380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:51.102 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:51.100+0000 7f8ab8f43380 1 bucket sync caught up with source: 2021-09-17T02:02:51.102 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , 00000000004.256.9, , , 00000000065.423.9, , ] 2021-09-17T02:02:51.103 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , 00000000004.256.9, , , 00000000065.423.9, , ] 2021-09-17T02:02:51.103 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:02:51.100+0000 7f8ab8f43380 0 bucket checkpoint complete 2021-09-17T02:02:51.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:51.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:02:51.116 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:02:51.180 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:02:51.181 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:02:51.200 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:51.198+0000 7fb8481f0380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:02:51.200 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:51.198+0000 7fb8481f0380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:02:51.200 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:51.198+0000 7fb8481f0380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:02:51.200 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:51.198+0000 7fb8481f0380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:02:51.226 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:51.225+0000 7fb8481f0380 1 bucket sync caught up with source: 2021-09-17T02:02:51.227 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , 00000000004.256.9, , , 00000000065.423.9, , ] 2021-09-17T02:02:51.227 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , 00000000004.256.9, , , 00000000065.423.9, , ] 2021-09-17T02:02:51.227 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:02:51.225+0000 7fb8481f0380 0 bucket checkpoint complete 2021-09-17T02:02:51.241 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-19 zones={test-zone2, test-zone1} 2021-09-17T02:02:51.248 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:51.255 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:51.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.280 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:51.287 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:51.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:51.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:51.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:51.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:51.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:51.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:51.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:51.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:51.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:51.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:51.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:52.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:52.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:52.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:52.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:52.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:52.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:52.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:52.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:52.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:52.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:52.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:53.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:53.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:53.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:53.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:53.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:53.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:53.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:53.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:53.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:53.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:53.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:54.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:55.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:56.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:57.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:57.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:57.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:57.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:57.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:57.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:57.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:57.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:57.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:57.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:57.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:58.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:59.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:59.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:59.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-19 zones={test-zone2, test-zone1} 2021-09-17T02:02:59.242 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-19 zones={test-zone3, test-zone1} 2021-09-17T02:02:59.248 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:02:59.253 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:59.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.267 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:02:59.272 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:02:59.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:02:59.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:59.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:59.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:02:59.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:59.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:59.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:59.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:59.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:59.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:59.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:02:59.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:00.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:01.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:01.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:01.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:01.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:01.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:01.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:01.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:01.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:01.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:01.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:01.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:02.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:03.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:03.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:03.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:03.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:03.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:03.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:03.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:03.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:03.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:03.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:03.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:04.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:05.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:06.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:07.103 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-19 zones={test-zone3, test-zone1} 2021-09-17T02:03:07.103 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-19 zones={test-zone3, test-zone2} 2021-09-17T02:03:07.110 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:03:07.156 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:03:07.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.170 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:03:07.176 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T02:03:07.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:07.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:03:07.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:03:07.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T02:03:07.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:07.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:07.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:07.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:07.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:07.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:07.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:07.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:07.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:08.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:08.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:08.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:08.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:08.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:08.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:08.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:08.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:08.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:08.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:08.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:09.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:10.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:11.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:11.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:11.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:11.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:11.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:11.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:11.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:11.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:11.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:11.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:11.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:12.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:13.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:13.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:13.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:13.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:13.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:13.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:13.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:13.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:13.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:13.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:13.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:14.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:15.079 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-19 zones={test-zone3, test-zone2} 2021-09-17T02:03:15.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:15.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:15.081 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:15.131 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:15.131 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:15.192 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:03:15.192 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:15.192 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:15.193 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:03:05.903813Z" 2021-09-17T02:03:15.193 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:15.193 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:15.193 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:15.194 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:02:43.902112Z" 2021-09-17T02:03:15.194 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:15.194 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:15.194 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:15.194 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:15.195 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:15.195 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:15.195 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:15.195 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:02:21.900654Z" 2021-09-17T02:03:15.196 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:15.196 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:03:15.197 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-17T02:03:15.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:15.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:15.199 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:15.250 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:15.251 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:15.308 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:15.308 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:15.309 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:15.309 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:15.309 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:15.309 INFO:teuthology.orchestra.run.smithi045.stdout: "instance_id": 13741153044533004467 2021-09-17T02:03:15.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:15.310 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:15.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:15.310 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:15.311 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:15.311 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:15.311 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000000911", 2021-09-17T02:03:15.311 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:15.312 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:15.312 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:15.312 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:44.029603Z" 2021-09-17T02:03:15.312 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:15.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:15.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:15.313 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:15.313 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:15.315 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:15.315 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:15.316 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:15.316 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:15.316 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:15.317 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:43.902112Z" 2021-09-17T02:03:15.317 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:15.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:15.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:15.318 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:15.318 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:15.318 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:15.318 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:15.318 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:15.319 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:15.319 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:15.319 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000" 2021-09-17T02:03:15.319 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:15.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:15.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:15.320 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:15.320 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:15.320 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:15.321 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:15.321 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:15.321 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:15.321 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:15.322 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:21.900654Z" 2021-09-17T02:03:15.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:15.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:15.322 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:15.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:15.323 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:15.323 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 0, 2021-09-17T02:03:15.323 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 0 2021-09-17T02:03:15.323 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:15.324 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:15.325 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000001044 target=00000000000000000000:00000000000000000911 2021-09-17T02:03:20.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:20.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:20.329 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:20.381 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:20.382 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:20.439 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:20.439 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:20.439 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:20.440 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:20.440 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:20.440 INFO:teuthology.orchestra.run.smithi045.stdout: "instance_id": 13741153044533004467 2021-09-17T02:03:20.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:20.441 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:20.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:20.441 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:20.441 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:20.441 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:20.442 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000000911", 2021-09-17T02:03:20.442 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:20.442 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:20.442 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:20.443 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:44.029603Z" 2021-09-17T02:03:20.443 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:20.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:20.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:20.444 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:20.444 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:20.444 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:20.444 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:20.444 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:20.445 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:20.445 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:20.445 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:43.902112Z" 2021-09-17T02:03:20.445 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:20.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:20.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:20.446 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:20.446 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:20.447 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:20.447 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:20.447 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:20.447 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:20.447 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:20.448 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000" 2021-09-17T02:03:20.448 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:20.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:20.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:20.449 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:20.449 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:20.449 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:20.450 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:20.450 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:20.450 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:20.450 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:20.451 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:21.900654Z" 2021-09-17T02:03:20.451 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:20.451 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:20.451 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:20.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:20.452 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:20.452 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 0, 2021-09-17T02:03:20.452 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 0 2021-09-17T02:03:20.452 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:20.453 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:20.454 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000001044 target=00000000000000000000:00000000000000000911 2021-09-17T02:03:25.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:25.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:25.457 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:25.510 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:25.567 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:25.568 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:25.569 INFO:teuthology.orchestra.run.smithi045.stdout: "instance_id": 13741153044533004467 2021-09-17T02:03:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:25.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:25.572 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:25.573 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:05.903813Z" 2021-09-17T02:03:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:25.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:25.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:25.575 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:25.575 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:25.576 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:25.576 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:25.576 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:25.577 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:25.577 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:25.577 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:43.902112Z" 2021-09-17T02:03:25.578 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:25.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:25.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:25.579 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:25.579 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:25.579 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:25.580 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:25.580 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:25.580 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:25.581 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:25.581 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000" 2021-09-17T02:03:25.581 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:25.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:25.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:25.583 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:25.583 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:25.583 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:25.584 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:25.584 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:25.584 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:25.585 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:25.585 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:21.900654Z" 2021-09-17T02:03:25.585 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:25.586 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:25.586 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:25.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:25.587 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:25.587 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 0, 2021-09-17T02:03:25.587 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 0 2021-09-17T02:03:25.588 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:25.588 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:25.590 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-17T02:03:25.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:25.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:25.592 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:25.642 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:25.642 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:25.704 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:03:25.704 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.705 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:25.705 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:03:05.903813Z" 2021-09-17T02:03:25.705 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:25.705 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.706 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:25.706 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:02:43.902112Z" 2021-09-17T02:03:25.706 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:25.706 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.707 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:25.707 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:25.707 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:25.707 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.708 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:25.708 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:02:21.900654Z" 2021-09-17T02:03:25.708 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:25.708 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:03:25.709 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-17T02:03:25.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:25.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:25.711 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:25.757 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:25.758 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:25.807 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:03:25.807 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:03:25.808 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:03:25.808 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:03:25.808 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:03:25.808 INFO:teuthology.orchestra.run.smithi013.stdout: "instance_id": 7207807225462341899 2021-09-17T02:03:25.809 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:25.809 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:03:25.809 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.809 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:03:25.809 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:25.810 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:25.810 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:25.810 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:25.811 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:25.811 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:25.811 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:03:05.903813Z" 2021-09-17T02:03:25.812 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:25.812 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:25.812 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.812 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:03:25.812 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:25.813 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:25.813 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:25.813 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:25.813 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:25.813 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:25.814 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:02:43.902112Z" 2021-09-17T02:03:25.814 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:25.814 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:25.815 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.815 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:03:25.815 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:25.816 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:25.816 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:25.816 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:25.816 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:25.817 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:25.817 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000" 2021-09-17T02:03:25.817 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:25.817 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:25.817 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.818 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:03:25.818 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:25.818 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:25.818 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:25.819 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:25.819 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:25.819 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:25.819 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:02:21.900654Z" 2021-09-17T02:03:25.819 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:25.820 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:25.820 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:03:25.820 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:25.820 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:03:25.821 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 0, 2021-09-17T02:03:25.821 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 0 2021-09-17T02:03:25.821 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:25.822 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:03:25.823 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-17T02:03:25.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:25.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:25.825 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:25.867 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:25.867 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:25.920 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:03:25.921 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.921 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:25.921 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:03:05.903813Z" 2021-09-17T02:03:25.922 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:25.922 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.922 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:25.922 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:02:43.902112Z" 2021-09-17T02:03:25.923 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:25.923 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.923 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:25.923 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:25.924 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:25.924 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:25.925 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:25.925 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:02:21.900654Z" 2021-09-17T02:03:25.925 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:25.926 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:03:25.928 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-17T02:03:25.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:25.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:25.930 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:25.984 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:25.984 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:26.039 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:26.039 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:26.040 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:26.040 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:26.041 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:26.041 INFO:teuthology.orchestra.run.smithi045.stdout: "instance_id": 9108083374096008638 2021-09-17T02:03:26.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.042 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:26.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.043 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:26.043 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:26.044 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.044 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:26.045 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:26.045 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:26.046 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:26.046 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:05.903813Z" 2021-09-17T02:03:26.046 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.048 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:26.048 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:26.048 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.048 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:26.049 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:26.049 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:26.049 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:26.049 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:43.902112Z" 2021-09-17T02:03:26.049 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.050 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:26.050 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:26.051 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.051 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:26.051 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:26.051 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:26.051 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:26.052 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000" 2021-09-17T02:03:26.052 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.052 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:26.053 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:26.053 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.053 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:26.053 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:26.054 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:26.054 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:26.054 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:21.900654Z" 2021-09-17T02:03:26.054 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.055 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.055 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:26.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.055 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:26.055 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 0, 2021-09-17T02:03:26.056 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 0 2021-09-17T02:03:26.056 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.056 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:26.058 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-17T02:03:26.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.060 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:26.104 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:26.104 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:26.158 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2021-09-17T02:03:26.158 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.158 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:26.159 INFO:teuthology.orchestra.run.smithi045.stdout: "last_update": "2021-09-17T02:03:05.740814Z" 2021-09-17T02:03:26.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.159 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.159 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:26.160 INFO:teuthology.orchestra.run.smithi045.stdout: "last_update": "2021-09-17T02:02:43.739443Z" 2021-09-17T02:03:26.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.160 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:26.161 INFO:teuthology.orchestra.run.smithi045.stdout: "last_update": "0.000000" 2021-09-17T02:03:26.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.161 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.161 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:26.162 INFO:teuthology.orchestra.run.smithi045.stdout: "last_update": "2021-09-17T02:02:21.737905Z" 2021-09-17T02:03:26.162 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.162 INFO:teuthology.orchestra.run.smithi045.stdout:] 2021-09-17T02:03:26.163 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-17T02:03:26.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.165 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:26.213 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:26.213 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:26.267 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:03:26.267 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:03:26.268 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:03:26.268 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:03:26.268 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:03:26.268 INFO:teuthology.orchestra.run.smithi013.stdout: "instance_id": 13041151301434725975 2021-09-17T02:03:26.269 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.269 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:03:26.269 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.269 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:03:26.269 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.270 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.270 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:26.270 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.270 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.271 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.271 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:03:05.740814Z" 2021-09-17T02:03:26.271 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.271 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.271 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.272 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:03:26.272 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.272 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.272 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:26.273 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.273 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.273 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.273 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:02:43.739443Z" 2021-09-17T02:03:26.273 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.274 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.274 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.274 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:03:26.274 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.275 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.275 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:26.275 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.275 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.276 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.276 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000" 2021-09-17T02:03:26.276 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.276 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.276 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.277 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:03:26.277 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.277 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.277 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:26.278 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.278 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.278 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.278 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:02:21.737905Z" 2021-09-17T02:03:26.279 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.279 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.279 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:03:26.279 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.279 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:03:26.280 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 0, 2021-09-17T02:03:26.280 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 0 2021-09-17T02:03:26.280 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.280 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:03:26.281 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-17T02:03:26.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.283 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:26.326 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:26.378 INFO:teuthology.orchestra.run.smithi045.stdout:[ 2021-09-17T02:03:26.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.379 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:26.379 INFO:teuthology.orchestra.run.smithi045.stdout: "last_update": "2021-09-17T02:03:05.740814Z" 2021-09-17T02:03:26.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.380 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:26.380 INFO:teuthology.orchestra.run.smithi045.stdout: "last_update": "2021-09-17T02:02:43.739443Z" 2021-09-17T02:03:26.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.381 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:26.381 INFO:teuthology.orchestra.run.smithi045.stdout: "last_update": "0.000000" 2021-09-17T02:03:26.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.381 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:26.382 INFO:teuthology.orchestra.run.smithi045.stdout: "last_update": "2021-09-17T02:02:21.737905Z" 2021-09-17T02:03:26.382 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.382 INFO:teuthology.orchestra.run.smithi045.stdout:] 2021-09-17T02:03:26.385 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-17T02:03:26.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.386 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:26.435 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:26.435 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:26.484 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:03:26.485 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:03:26.485 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:03:26.485 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:03:26.486 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:03:26.486 INFO:teuthology.orchestra.run.smithi013.stdout: "instance_id": 16952598883558957051 2021-09-17T02:03:26.486 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.486 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:03:26.487 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.487 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:03:26.487 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.487 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.488 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:26.488 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.488 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.488 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.488 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:03:05.740814Z" 2021-09-17T02:03:26.489 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.489 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.489 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.489 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:03:26.490 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.490 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.490 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:26.490 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.490 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.491 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.491 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:02:43.739443Z" 2021-09-17T02:03:26.491 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.491 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.491 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.492 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:03:26.492 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.492 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.492 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:26.493 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.493 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.493 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.493 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000" 2021-09-17T02:03:26.493 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.494 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.494 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.494 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:03:26.495 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.495 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.495 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:26.495 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.496 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.496 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.496 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:02:21.737905Z" 2021-09-17T02:03:26.496 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.496 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.497 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:03:26.497 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.497 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:03:26.497 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 0, 2021-09-17T02:03:26.497 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 0 2021-09-17T02:03:26.498 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.498 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:03:26.499 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-17T02:03:26.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.501 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:26.544 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:26.544 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:26.597 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:03:26.597 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.597 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:26.597 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:03:05.714711Z" 2021-09-17T02:03:26.598 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.598 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.598 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:26.598 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:02:43.713407Z" 2021-09-17T02:03:26.599 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.599 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.599 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:26.600 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:26.600 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.600 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.600 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:26.601 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:02:21.711883Z" 2021-09-17T02:03:26.601 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.601 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:03:26.603 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-17T02:03:26.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.604 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:26.688 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:26.689 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:26.739 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:03:26.739 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:03:26.740 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:03:26.740 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:03:26.740 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:03:26.741 INFO:teuthology.orchestra.run.smithi013.stdout: "instance_id": 2421114679875278012 2021-09-17T02:03:26.741 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.741 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:03:26.741 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.741 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:03:26.742 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.742 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.742 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:26.742 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.743 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.743 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.743 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:03:05.714711Z" 2021-09-17T02:03:26.743 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.743 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.744 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.744 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:03:26.744 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.744 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.745 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:26.745 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.745 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.745 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.745 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:02:43.713407Z" 2021-09-17T02:03:26.746 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.746 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.746 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.746 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:03:26.747 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.747 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.747 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:26.747 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.747 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.748 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.748 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000" 2021-09-17T02:03:26.748 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.748 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.749 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.749 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:03:26.749 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:26.749 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.749 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:26.750 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:26.750 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:26.750 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:26.750 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:02:21.711883Z" 2021-09-17T02:03:26.750 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.751 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.751 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:03:26.751 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.751 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:03:26.752 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 0, 2021-09-17T02:03:26.752 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 0 2021-09-17T02:03:26.752 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.752 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:03:26.753 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-17T02:03:26.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.755 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:26.797 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:26.798 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:26.850 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:03:26.850 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.851 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:26.851 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:03:05.714711Z" 2021-09-17T02:03:26.851 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.851 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.852 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:26.852 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:02:43.713407Z" 2021-09-17T02:03:26.852 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.852 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.853 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:26.853 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:26.853 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:26.853 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:26.854 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:26.854 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:02:21.711883Z" 2021-09-17T02:03:26.854 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:26.854 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:03:26.855 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-17T02:03:26.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.857 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:26.905 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:26.956 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:26.957 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:26.957 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:26.957 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:26.957 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:26.958 INFO:teuthology.orchestra.run.smithi045.stdout: "instance_id": 12700487463353247569 2021-09-17T02:03:26.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.958 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:26.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.959 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:26.959 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:26.960 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.960 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T02:03:26.961 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:26.961 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:26.961 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:26.962 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:05.714711Z" 2021-09-17T02:03:26.962 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.964 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:26.964 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:26.965 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.965 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000003704", 2021-09-17T02:03:26.965 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:26.966 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:26.966 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:26.967 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:43.713407Z" 2021-09-17T02:03:26.967 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.969 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:26.969 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:26.969 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.970 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:26.970 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:26.971 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:26.971 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:26.972 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000" 2021-09-17T02:03:26.972 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:26.973 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:26.974 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:26.974 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "incremental-sync", 2021-09-17T02:03:26.975 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-17T02:03:26.975 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:26.975 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:26.976 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:26.976 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:02:21.711883Z" 2021-09-17T02:03:26.976 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.976 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.977 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:26.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:26.977 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:26.977 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 0, 2021-09-17T02:03:26.977 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 0 2021-09-17T02:03:26.978 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:26.978 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:26.980 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-17T02:03:26.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-09-17T02:03:26.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:26.997 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:27.041 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:27.041 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:27.077 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:03:27.091 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:03:27.091 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:27.091 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:27.091 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:03:27.092 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:27.092 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:27.092 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844206.994306_186.1", 2021-09-17T02:03:27.092 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:03:26.994306Z" 2021-09-17T02:03:27.092 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:27.093 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:27.093 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:27.093 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:27.093 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:27.094 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:27.094 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:27.094 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:27.094 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:27.095 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:03:27.096 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:03:27.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:27.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:27.097 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:27.141 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:27.142 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:27.191 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:27.192 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:27.192 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:27.192 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:27.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:27.193 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:27.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:27.193 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:27.193 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:27.193 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:27.194 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:27.194 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:27.194 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:03:27.194 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:27.194 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:27.195 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:27.195 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:27.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:27.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:27.195 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:27.196 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:27.196 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:27.196 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844206.988214_183.1", 2021-09-17T02:03:27.196 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:27.196 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:27.197 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:27.197 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:26.988214Z", 2021-09-17T02:03:27.197 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:27.197 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:27.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:27.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:27.198 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:27.198 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:27.198 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:27.199 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:27.199 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:27.199 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:27.199 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:27.199 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:27.200 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:27.200 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:27.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:27.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:27.200 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:27.201 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:27.201 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:27.201 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:27.201 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:27.201 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:27.202 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:27.202 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:27.202 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:27.202 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:27.202 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:27.203 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:27.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:27.203 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:27.203 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:03:27.204 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:03:27.204 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:27.204 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:27.205 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844206.994306_186.1 target=1_1631844206.988214_183.1 2021-09-17T02:03:32.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:32.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:32.208 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:32.258 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:32.313 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:32.314 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:32.314 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:32.314 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:32.314 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:32.315 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:32.315 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:32.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:32.315 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:32.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:32.316 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:32.316 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:32.316 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:32.317 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:32.317 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:32.317 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:03:32.317 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:32.317 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:32.318 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:32.318 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:32.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:32.319 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:32.319 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:32.319 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:32.319 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844206.994306_186.1", 2021-09-17T02:03:32.319 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:32.320 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:32.320 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:32.320 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:26.994306Z", 2021-09-17T02:03:32.320 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:32.320 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:32.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:32.321 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:32.321 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:32.322 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:32.322 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:32.322 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:32.322 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:32.322 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:32.323 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:32.323 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:32.323 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:32.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:32.324 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:32.324 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:32.324 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:32.324 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:32.325 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:32.325 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:32.325 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:32.325 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:32.325 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:32.326 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.326 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.326 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:32.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:32.326 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:32.327 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:03:32.327 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:03:32.327 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.327 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:32.329 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:03:32.329 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:03:32.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:32.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:32.331 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:32.380 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:32.380 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:32.435 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:03:32.436 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:03:32.436 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:03:32.436 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:03:32.437 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:03:32.437 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:32.437 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:32.437 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:32.438 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:03:32.438 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:32.438 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:03:32.438 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:32.438 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:32.439 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:32.439 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:32.439 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:03:32.439 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:32.440 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:32.440 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:32.440 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:32.440 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:32.440 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:32.441 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:03:32.441 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:32.441 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:32.441 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844206.994306_186.1", 2021-09-17T02:03:32.442 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:32.442 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:32.442 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:32.442 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:03:26.994306Z", 2021-09-17T02:03:32.442 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:32.443 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:32.443 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:32.443 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:32.443 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:03:32.443 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:32.444 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:32.444 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:32.444 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:32.444 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:32.445 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:32.445 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:03:32.445 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:32.445 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:32.445 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:32.446 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:32.446 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:03:32.446 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:32.446 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:32.446 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:32.447 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:32.447 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:32.447 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:32.447 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:03:32.447 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:32.448 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:32.448 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:32.448 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:03:32.448 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:32.448 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:03:32.449 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:03:32.449 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:03:32.449 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:32.449 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:03:32.451 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:03:32.451 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:03:32.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:32.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:32.453 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:32.499 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:32.499 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:32.551 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:32.552 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:32.552 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:32.552 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:32.553 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:32.553 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:32.553 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:32.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:32.553 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:32.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:32.554 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:32.554 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:32.554 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:32.555 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:32.555 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:32.555 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:03:32.555 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:32.556 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:32.556 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:32.556 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:32.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:32.557 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:32.557 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:32.557 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:32.557 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844206.994306_186.1", 2021-09-17T02:03:32.558 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:32.558 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:32.558 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:32.558 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:26.994306Z", 2021-09-17T02:03:32.559 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:32.559 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:32.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:32.560 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:32.560 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:32.560 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:32.560 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:32.560 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:32.561 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:32.561 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:32.561 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:32.561 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:32.561 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:32.562 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:32.563 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:32.564 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:32.564 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:32.564 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:32.564 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.565 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.565 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:32.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:32.565 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:03:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:03:32.566 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:32.566 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:32.568 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:03:32.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:32.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:32.617 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:32.686 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:32.684+0000 7fa3fb945380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:32.687 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:32.684+0000 7fa3fb945380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:32.687 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:32.684+0000 7fa3fb945380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:03:32.687 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:32.684+0000 7fa3fb945380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:32.712 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:32.710+0000 7fa3fb945380 1 waiting for incremental sync to catch up: 2021-09-17T02:03:32.712 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , , , ] 2021-09-17T02:03:32.712 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000001.9.5, , ] 2021-09-17T02:03:37.713 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:37.712+0000 7fa3fb945380 1 bucket sync caught up with source: 2021-09-17T02:03:37.714 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000001.9.5, , ] 2021-09-17T02:03:37.714 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000001.9.5, , ] 2021-09-17T02:03:37.714 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:37.712+0000 7fa3fb945380 0 bucket checkpoint complete 2021-09-17T02:03:37.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:37.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:37.731 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:37.789 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:37.789 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:37.810 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:37.810+0000 7f9393317380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:37.810 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:37.810+0000 7f9393317380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:37.810 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:37.810+0000 7f9393317380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:03:37.811 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:37.810+0000 7f9393317380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:37.837 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:37.836+0000 7f9393317380 1 bucket sync caught up with source: 2021-09-17T02:03:37.837 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.9.5, , ] 2021-09-17T02:03:37.837 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.9.5, , ] 2021-09-17T02:03:37.838 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:37.836+0000 7f9393317380 0 bucket checkpoint complete 2021-09-17T02:03:37.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:37.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:37.852 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:37.909 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:37.909 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:37.930 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:37.929+0000 7f9dba4ea380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:37.931 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:37.929+0000 7f9dba4ea380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:37.931 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:37.929+0000 7f9dba4ea380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:03:37.931 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:37.929+0000 7f9dba4ea380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:37.957 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:37.956+0000 7f9dba4ea380 1 bucket sync caught up with source: 2021-09-17T02:03:37.958 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000001.9.5, , ] 2021-09-17T02:03:37.958 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000001.9.5, , ] 2021-09-17T02:03:37.958 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:37.956+0000 7f9dba4ea380 0 bucket checkpoint complete 2021-09-17T02:03:37.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:37.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:37.972 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:38.024 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:38.024 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:38.043 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.042+0000 7f39be94f380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:38.043 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.042+0000 7f39be94f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:38.043 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.042+0000 7f39be94f380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:03:38.043 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.042+0000 7f39be94f380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:38.068 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.068+0000 7f39be94f380 1 bucket sync caught up with source: 2021-09-17T02:03:38.068 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.183.5, , ] 2021-09-17T02:03:38.068 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.183.5, , ] 2021-09-17T02:03:38.069 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.068+0000 7f39be94f380 0 bucket checkpoint complete 2021-09-17T02:03:38.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:38.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:38.083 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:38.135 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:38.135 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:38.153 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.152+0000 7fb39ea95380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:38.153 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.152+0000 7fb39ea95380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:38.153 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.152+0000 7fb39ea95380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:03:38.154 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.152+0000 7fb39ea95380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:38.178 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.177+0000 7fb39ea95380 1 bucket sync caught up with source: 2021-09-17T02:03:38.178 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.183.5, , ] 2021-09-17T02:03:38.179 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.183.5, , ] 2021-09-17T02:03:38.179 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.177+0000 7fb39ea95380 0 bucket checkpoint complete 2021-09-17T02:03:38.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:38.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:38.192 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:38.244 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:38.245 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:38.263 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.263+0000 7fa113dc1380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:38.264 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.263+0000 7fa113dc1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:38.264 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.263+0000 7fa113dc1380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:03:38.264 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.263+0000 7fa113dc1380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:38.288 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.288+0000 7fa113dc1380 1 bucket sync caught up with source: 2021-09-17T02:03:38.289 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.205.5, , ] 2021-09-17T02:03:38.289 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.205.5, , ] 2021-09-17T02:03:38.289 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:38.288+0000 7fa113dc1380 0 bucket checkpoint complete 2021-09-17T02:03:38.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:38.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:38.304 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:38.360 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:38.360 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:38.380 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:38.379+0000 7f6119e4e380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:38.381 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:38.379+0000 7f6119e4e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:38.381 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:38.379+0000 7f6119e4e380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:03:38.381 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:38.379+0000 7f6119e4e380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:38.405+0000 7f6119e4e380 1 bucket sync caught up with source: 2021-09-17T02:03:38.407 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000001.205.5, , ] 2021-09-17T02:03:38.408 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000001.205.5, , ] 2021-09-17T02:03:38.408 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:38.405+0000 7f6119e4e380 0 bucket checkpoint complete 2021-09-17T02:03:38.422 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-20 zones={test-zone2, test-zone1} 2021-09-17T02:03:38.429 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:03:38.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:38.480 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:03:38.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:38.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:38.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:38.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:38.650 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-20 zones={test-zone2, test-zone1} 2021-09-17T02:03:38.650 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-20 zones={test-zone3, test-zone1} 2021-09-17T02:03:38.655 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:03:38.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:38.700 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:03:38.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:38.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:38.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:38.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:38.862 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-20 zones={test-zone3, test-zone1} 2021-09-17T02:03:38.863 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-20 zones={test-zone3, test-zone2} 2021-09-17T02:03:38.867 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:03:38.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:38.912 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:03:38.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:38.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:38.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:39.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:39.082 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-20 zones={test-zone3, test-zone2} 2021-09-17T02:03:39.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:39.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:39.089 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:39.133 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:39.133 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:39.170 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:03:39.182 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:03:39.183 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:39.183 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:39.183 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:03:39.183 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:39.184 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:39.184 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844219.087460_188.1", 2021-09-17T02:03:39.184 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:03:39.087460Z" 2021-09-17T02:03:39.184 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:39.185 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:39.185 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:39.185 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:39.185 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:39.186 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:39.186 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:39.186 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:39.186 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:39.187 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:03:39.188 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:03:39.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:39.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:39.190 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:39.241 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:39.241 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:39.296 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:39.297 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:39.297 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:39.298 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:39.298 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:39.298 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:39.299 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:39.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:39.299 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:39.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:39.300 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:39.300 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:39.300 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:39.300 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:39.300 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:39.301 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:03:39.301 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:39.301 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:39.301 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:39.301 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:39.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:39.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:39.302 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:39.302 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:39.303 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:39.303 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844219.084690_187.1", 2021-09-17T02:03:39.303 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:39.303 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:39.303 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:39.304 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:39.084690Z", 2021-09-17T02:03:39.304 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:39.304 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:39.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:39.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:39.305 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:39.305 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:39.305 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:39.305 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:39.306 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:39.306 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:39.306 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:39.306 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:39.307 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:39.307 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:39.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:39.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:39.308 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:39.308 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:39.309 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:39.310 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:39.310 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:39.312 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:39.312 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:39.312 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:39.312 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:39.313 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:39.313 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:39.313 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:39.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:39.314 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:39.314 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:03:39.314 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:03:39.314 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:39.315 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:39.316 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844219.087460_188.1 target=1_1631844219.084690_187.1 2021-09-17T02:03:44.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:44.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:44.319 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:44.370 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:44.370 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:44.425 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:44.425 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:44.426 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:44.426 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:44.426 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:44.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:44.427 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:44.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:44.427 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:44.428 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:44.428 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:44.428 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:44.428 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:44.429 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:03:44.429 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:44.429 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:44.430 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:44.430 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:44.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:44.431 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:44.431 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:44.432 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:44.432 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844219.087460_188.1", 2021-09-17T02:03:44.432 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:44.433 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:44.433 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:44.433 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:39.087460Z", 2021-09-17T02:03:44.434 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:44.434 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:44.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:44.434 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:44.435 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:44.435 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:44.435 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:44.435 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:44.436 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:44.436 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:44.436 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:44.436 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:44.436 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:44.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:44.437 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:44.437 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:44.437 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:44.438 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:44.438 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:44.438 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:44.438 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:44.438 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:44.439 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:44.439 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.439 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.439 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:44.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:44.440 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:44.440 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:03:44.440 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:03:44.440 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.441 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:44.442 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:03:44.443 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:03:44.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:44.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:44.444 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:44.495 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:44.495 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:44.550 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:03:44.550 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:03:44.551 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:03:44.551 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:03:44.551 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:03:44.551 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:44.552 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:44.552 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:44.552 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:03:44.552 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:44.552 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:03:44.554 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:44.554 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:44.554 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:44.555 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:44.555 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:03:44.555 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:44.555 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:44.556 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:44.556 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:44.556 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:44.556 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:44.556 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:03:44.557 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:44.557 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:44.557 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844219.087460_188.1", 2021-09-17T02:03:44.557 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:44.558 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:44.558 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:44.558 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:03:39.087460Z", 2021-09-17T02:03:44.558 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:44.559 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:44.559 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:44.559 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:44.559 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:03:44.560 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:44.560 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:44.560 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:44.561 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:44.561 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:44.561 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:44.561 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:03:44.561 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:44.562 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:44.562 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:44.562 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:44.562 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:03:44.563 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:44.563 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:44.563 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:44.563 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:44.564 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:44.564 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:44.564 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:03:44.564 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:44.564 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:44.565 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:44.565 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:03:44.565 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:44.565 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:03:44.566 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:03:44.566 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:03:44.566 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:44.566 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:03:44.567 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:03:44.568 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:03:44.568 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:44.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:44.569 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:44.615 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:44.616 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:44.666 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:44.666 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:44.667 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:44.667 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:44.667 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:44.668 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:44.668 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:44.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:44.668 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:44.669 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:03:44.670 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:44.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:44.672 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:44.672 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:44.672 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:44.672 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844219.087460_188.1", 2021-09-17T02:03:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:39.087460Z", 2021-09-17T02:03:44.673 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:44.674 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:44.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:44.674 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:44.675 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:44.676 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:44.676 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:44.676 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:44.676 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:44.676 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:44.677 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:44.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:44.677 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:44.677 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:44.678 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:44.678 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:44.678 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:44.678 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:44.679 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:44.679 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:44.679 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:44.680 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.680 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.680 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:44.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:44.681 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:44.681 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:03:44.682 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:03:44.682 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:44.682 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:44.684 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:03:44.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:44.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:44.733 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:44.785 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:44.785 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:44.804 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:44.802+0000 7ff544995380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:44.802+0000 7ff544995380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:44.805 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:44.802+0000 7ff544995380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:03:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:44.802+0000 7ff544995380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:44.829 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:44.827+0000 7ff544995380 1 bucket sync caught up with source: 2021-09-17T02:03:44.830 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.12.12, , ] 2021-09-17T02:03:44.830 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000003.12.12, , ] 2021-09-17T02:03:44.833 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:44.827+0000 7ff544995380 0 bucket checkpoint complete 2021-09-17T02:03:44.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:44.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:44.842 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:44.895 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:44.896 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:44.917 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:44.917+0000 7ffa41b56380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:44.918 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:44.917+0000 7ffa41b56380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:44.918 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:44.917+0000 7ffa41b56380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:03:44.918 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:44.917+0000 7ffa41b56380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:44.943 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:44.942+0000 7ffa41b56380 1 bucket sync caught up with source: 2021-09-17T02:03:44.943 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000003.12.12, , ] 2021-09-17T02:03:44.943 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000003.12.12, , ] 2021-09-17T02:03:44.944 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:44.942+0000 7ffa41b56380 0 bucket checkpoint complete 2021-09-17T02:03:44.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:44.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:44.957 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:45.007 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:45.008 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:45.026 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.024+0000 7fdafbd0e380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:45.026 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.024+0000 7fdafbd0e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:45.027 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.024+0000 7fdafbd0e380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:03:45.027 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.024+0000 7fdafbd0e380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:45.050 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.048+0000 7fdafbd0e380 1 bucket sync caught up with source: 2021-09-17T02:03:45.050 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.12.12, , ] 2021-09-17T02:03:45.051 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000003.12.12, , ] 2021-09-17T02:03:45.051 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.048+0000 7fdafbd0e380 0 bucket checkpoint complete 2021-09-17T02:03:45.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:45.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:45.064 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:45.115 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:45.115 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:45.133 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.133+0000 7f86ee293380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:45.134 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.133+0000 7f86ee293380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:45.134 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.133+0000 7f86ee293380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:03:45.134 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.133+0000 7f86ee293380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:45.159 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.158+0000 7f86ee293380 1 bucket sync caught up with source: 2021-09-17T02:03:45.159 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000004.208.9, , ] 2021-09-17T02:03:45.160 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000004.208.9, , ] 2021-09-17T02:03:45.160 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.158+0000 7f86ee293380 0 bucket checkpoint complete 2021-09-17T02:03:45.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:45.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:45.173 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:45.223 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:45.224 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:45.242 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.241+0000 7fb87c7e9380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:45.242 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.241+0000 7fb87c7e9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:45.243 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.241+0000 7fb87c7e9380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:03:45.243 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.241+0000 7fb87c7e9380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:45.267 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.266+0000 7fb87c7e9380 1 bucket sync caught up with source: 2021-09-17T02:03:45.267 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000004.208.9, , ] 2021-09-17T02:03:45.267 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000004.208.9, , ] 2021-09-17T02:03:45.267 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.266+0000 7fb87c7e9380 0 bucket checkpoint complete 2021-09-17T02:03:45.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:45.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:45.281 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:45.333 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:45.333 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:45.351 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.350+0000 7f6e285ed380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:45.351 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.350+0000 7f6e285ed380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:45.351 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.350+0000 7f6e285ed380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:03:45.351 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.350+0000 7f6e285ed380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:45.376 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.375+0000 7f6e285ed380 1 bucket sync caught up with source: 2021-09-17T02:03:45.376 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000004.230.9, , ] 2021-09-17T02:03:45.376 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000004.230.9, , ] 2021-09-17T02:03:45.376 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:45.375+0000 7f6e285ed380 0 bucket checkpoint complete 2021-09-17T02:03:45.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:45.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:45.390 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:45.443 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:45.443 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:45.461 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.460+0000 7f0a5f9e3380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.460+0000 7f0a5f9e3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.460+0000 7f0a5f9e3380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:03:45.462 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.460+0000 7f0a5f9e3380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:45.486 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.484+0000 7f0a5f9e3380 1 bucket sync caught up with source: 2021-09-17T02:03:45.487 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000004.230.9, , ] 2021-09-17T02:03:45.487 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000004.230.9, , ] 2021-09-17T02:03:45.487 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:45.484+0000 7f0a5f9e3380 0 bucket checkpoint complete 2021-09-17T02:03:45.499 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-20 zones={test-zone2, test-zone1} 2021-09-17T02:03:45.505 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:03:45.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:45.552 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:45.552 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:03:45.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:45.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:45.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:45.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:45.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:45.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:45.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:45.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:45.844 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-20 zones={test-zone2, test-zone1} 2021-09-17T02:03:45.845 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-20 zones={test-zone3, test-zone1} 2021-09-17T02:03:45.850 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:03:45.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:45.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:45.896 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:03:45.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:45.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:45.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:45.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:46.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:46.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:46.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:46.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:46.178 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-20 zones={test-zone3, test-zone1} 2021-09-17T02:03:46.178 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-20 zones={test-zone3, test-zone2} 2021-09-17T02:03:46.183 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:03:46.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:46.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:46.228 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:03:46.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:46.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:46.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:46.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:46.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:46.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:46.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:46.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:46.523 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-20 zones={test-zone3, test-zone2} 2021-09-17T02:03:46.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:46.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:46.530 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:46.573 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:46.574 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:46.610 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:03:46.622 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:03:46.622 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:46.623 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:46.623 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:03:46.623 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:46.623 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:46.624 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844226.528216_190.1", 2021-09-17T02:03:46.624 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:03:46.528216Z" 2021-09-17T02:03:46.624 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:46.624 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:46.625 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:46.625 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:46.625 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:46.625 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:46.625 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:46.626 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:46.626 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:46.626 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:03:46.627 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:03:46.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:46.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:46.629 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:46.678 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:46.680 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:46.735 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:46.735 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:46.735 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:46.736 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:46.736 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:46.736 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:46.736 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:46.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:46.737 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:46.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:46.737 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:46.738 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:46.738 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:46.738 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:46.739 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:46.739 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:03:46.739 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:46.740 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:46.740 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:46.740 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:46.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:46.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:46.741 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:46.741 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:46.741 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:46.741 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844226.525402_189.1", 2021-09-17T02:03:46.742 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:46.742 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:46.742 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:46.742 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:46.525402Z", 2021-09-17T02:03:46.743 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:46.743 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:46.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:46.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:46.744 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:46.744 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:46.744 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:46.744 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:46.745 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:46.745 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:46.745 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:46.745 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:46.746 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:46.746 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:46.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:46.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:46.747 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:46.747 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:46.747 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:46.748 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:46.748 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:46.748 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:46.749 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:46.749 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:46.749 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:46.749 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:46.750 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:46.750 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:46.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:46.750 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:46.751 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:03:46.751 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:03:46.751 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:46.751 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:46.753 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844226.528216_190.1 target=1_1631844226.525402_189.1 2021-09-17T02:03:51.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:51.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:51.755 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:51.806 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:51.807 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:51.861 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:51.861 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:51.861 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:51.862 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:51.862 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:51.862 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:51.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:51.864 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:51.864 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:51.864 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:03:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:51.867 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:51.867 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:51.867 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:51.867 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844226.528216_190.1", 2021-09-17T02:03:51.868 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:51.868 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:51.868 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:51.868 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:46.528216Z", 2021-09-17T02:03:51.868 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:51.869 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:51.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:51.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:51.869 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:51.870 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:51.870 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:51.870 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:51.870 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:51.870 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:51.871 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:51.871 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:51.871 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:51.871 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:51.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:51.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:51.872 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:51.872 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:51.872 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:51.873 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:51.873 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:51.873 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:51.873 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:51.874 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:51.874 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:51.874 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:51.874 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:51.875 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:51.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:51.875 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:51.875 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:03:51.875 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:03:51.876 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:51.876 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:51.877 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:03:51.878 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:03:51.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:51.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:51.880 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:51.930 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:51.930 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:51.985 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:03:51.985 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:03:51.986 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:03:51.986 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:03:51.986 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:03:51.986 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:51.987 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:51.987 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:51.987 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:03:51.987 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:51.988 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:03:51.988 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:51.988 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:51.988 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:51.989 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:51.989 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:03:51.989 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:51.989 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:51.990 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:51.990 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:51.990 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:51.990 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:51.991 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:03:51.991 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:51.991 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:51.991 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844226.528216_190.1", 2021-09-17T02:03:51.992 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:51.992 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:51.992 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:51.992 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:03:46.528216Z", 2021-09-17T02:03:51.992 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:51.993 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:51.993 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:51.993 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:51.993 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:03:51.994 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:51.994 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:51.994 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:51.994 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:51.995 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:51.995 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:51.995 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:03:51.995 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:51.995 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:51.996 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:51.996 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:51.996 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:03:51.996 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:03:51.996 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:03:51.997 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:51.997 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:03:51.997 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:03:51.998 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:03:51.998 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:03:51.998 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:03:51.998 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:51.998 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:51.999 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:03:51.999 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:51.999 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:03:51.999 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:03:52.000 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:03:52.000 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:52.000 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:03:52.001 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:03:52.002 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:03:52.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:52.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:52.003 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:52.103 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:52.103 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:52.104 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:52.104 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:52.105 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:52.105 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:52.106 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:52.106 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:52.106 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:52.106 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:52.107 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:52.107 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:52.107 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:52.108 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:52.108 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:03:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:52.109 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:52.110 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:52.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844226.528216_190.1", 2021-09-17T02:03:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:52.114 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:46.528216Z", 2021-09-17T02:03:52.115 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:52.116 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:52.117 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:52.118 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:52.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:52.120 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:52.121 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:52.121 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:52.121 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:52.121 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:52.121 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:52.122 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:52.122 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:52.122 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:52.122 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:52.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:52.123 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:52.123 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:03:52.123 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:03:52.123 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:52.124 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:52.125 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:03:52.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:52.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:52.172 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:52.225 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:52.226 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:52.245 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:52.243+0000 7f3896d4e380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:52.245 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:52.243+0000 7f3896d4e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:52.246 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:52.243+0000 7f3896d4e380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:03:52.246 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:52.243+0000 7f3896d4e380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:52.269 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:52.268+0000 7f3896d4e380 1 waiting for incremental sync to catch up: 2021-09-17T02:03:52.270 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.12.12, , ] 2021-09-17T02:03:52.270 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000005.16.17, , ] 2021-09-17T02:03:57.271 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.269+0000 7f3896d4e380 1 bucket sync caught up with source: 2021-09-17T02:03:57.271 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000005.16.17, , ] 2021-09-17T02:03:57.271 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000005.16.17, , ] 2021-09-17T02:03:57.272 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.269+0000 7f3896d4e380 0 bucket checkpoint complete 2021-09-17T02:03:57.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.289 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:57.346 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:57.347 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:57.368 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.367+0000 7f0394239380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:57.369 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.367+0000 7f0394239380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:57.369 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.367+0000 7f0394239380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:03:57.369 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.367+0000 7f0394239380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:57.396 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.395+0000 7f0394239380 1 bucket sync caught up with source: 2021-09-17T02:03:57.396 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000005.16.17, , ] 2021-09-17T02:03:57.396 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000005.16.17, , ] 2021-09-17T02:03:57.396 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.395+0000 7f0394239380 0 bucket checkpoint complete 2021-09-17T02:03:57.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.411 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:57.469 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:57.470 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:57.491 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.489+0000 7f858efd3380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:57.491 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.489+0000 7f858efd3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:57.491 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.489+0000 7f858efd3380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:03:57.492 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.489+0000 7f858efd3380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:57.518 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.516+0000 7f858efd3380 1 bucket sync caught up with source: 2021-09-17T02:03:57.518 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000005.16.17, , ] 2021-09-17T02:03:57.518 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000005.16.17, , ] 2021-09-17T02:03:57.519 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.516+0000 7f858efd3380 0 bucket checkpoint complete 2021-09-17T02:03:57.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.532 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:57.585 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:57.585 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:57.604 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.603+0000 7f8b10e71380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:57.605 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.603+0000 7f8b10e71380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:57.605 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.603+0000 7f8b10e71380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:03:57.605 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.603+0000 7f8b10e71380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:57.631 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.629+0000 7f8b10e71380 1 bucket sync caught up with source: 2021-09-17T02:03:57.631 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000007.234.14, , ] 2021-09-17T02:03:57.631 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000007.234.14, , ] 2021-09-17T02:03:57.631 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.629+0000 7f8b10e71380 0 bucket checkpoint complete 2021-09-17T02:03:57.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.645 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:57.694 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:57.695 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:57.713 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.711+0000 7f8c43072380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:57.713 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.711+0000 7f8c43072380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:57.713 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.711+0000 7f8c43072380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:03:57.713 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.711+0000 7f8c43072380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:57.737 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.736+0000 7f8c43072380 1 bucket sync caught up with source: 2021-09-17T02:03:57.737 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000007.234.14, , ] 2021-09-17T02:03:57.738 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000007.234.14, , ] 2021-09-17T02:03:57.738 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.736+0000 7f8c43072380 0 bucket checkpoint complete 2021-09-17T02:03:57.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.751 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:57.801 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:57.801 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:57.819 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.818+0000 7f0084e26380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:57.820 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.818+0000 7f0084e26380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:57.820 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.818+0000 7f0084e26380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:03:57.820 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.818+0000 7f0084e26380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:57.844 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.843+0000 7f0084e26380 1 bucket sync caught up with source: 2021-09-17T02:03:57.845 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000007.256.14, , ] 2021-09-17T02:03:57.845 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000007.256.14, , ] 2021-09-17T02:03:57.845 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:03:57.843+0000 7f0084e26380 0 bucket checkpoint complete 2021-09-17T02:03:57.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:03:57.859 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:03:57.914 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:57.914 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:57.940 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.933+0000 7f0c4fb07380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:03:57.940 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.933+0000 7f0c4fb07380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:03:57.940 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.933+0000 7f0c4fb07380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:03:57.941 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.933+0000 7f0c4fb07380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:03:57.962 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.961+0000 7f0c4fb07380 1 bucket sync caught up with source: 2021-09-17T02:03:57.962 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000007.256.14, , ] 2021-09-17T02:03:57.963 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000007.256.14, , ] 2021-09-17T02:03:57.963 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:03:57.961+0000 7f0c4fb07380 0 bucket checkpoint complete 2021-09-17T02:03:57.976 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-20 zones={test-zone2, test-zone1} 2021-09-17T02:03:57.981 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:03:58.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.028 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:03:58.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.301 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-20 zones={test-zone2, test-zone1} 2021-09-17T02:03:58.302 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-20 zones={test-zone3, test-zone1} 2021-09-17T02:03:58.307 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:03:58.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.352 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:03:58.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.633 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-20 zones={test-zone3, test-zone1} 2021-09-17T02:03:58.634 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-20 zones={test-zone3, test-zone2} 2021-09-17T02:03:58.638 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:03:58.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.684 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:03:58.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:03:58.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:03:58.970 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-20 zones={test-zone3, test-zone2} 2021-09-17T02:03:58.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-09-17T02:03:58.971 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-21 2021-09-17T02:03:58.986 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-22 2021-09-17T02:03:59.010 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-23 2021-09-17T02:03:59.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:59.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:59.056 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:59.098 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:59.099 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:59.143 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:03:59.157 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:03:59.158 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:59.158 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:59.158 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:03:59.159 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:59.159 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:59.159 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844239.051805_208.1", 2021-09-17T02:03:59.159 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:03:59.051805Z" 2021-09-17T02:03:59.159 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:59.160 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:59.160 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:59.160 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:59.160 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:03:59.160 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:03:59.161 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:03:59.161 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:03:59.161 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:03:59.161 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:03:59.162 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:03:59.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:59.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:03:59.164 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:03:59.210 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:03:59.211 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:03:59.262 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:03:59.262 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:03:59.262 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:03:59.263 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:03:59.263 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:03:59.263 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:03:59.263 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:59.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:59.264 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:03:59.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:59.264 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:03:59.264 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:59.264 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:59.265 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:03:59.265 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:59.265 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:03:59.265 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:59.266 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:03:59.266 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:59.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:59.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:59.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:59.267 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:03:59.267 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:59.267 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:59.267 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844238.977716_191.1", 2021-09-17T02:03:59.268 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:59.268 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:59.268 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:59.268 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:58.977716Z", 2021-09-17T02:03:59.268 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:59.269 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:59.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:59.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:59.269 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:03:59.269 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:59.270 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:59.270 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:59.270 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:59.270 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:59.271 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:59.271 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:59.271 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:59.271 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:59.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:59.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:03:59.272 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:03:59.272 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:03:59.272 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:03:59.272 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:03:59.273 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:03:59.273 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:03:59.273 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:03:59.273 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:03:59.273 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:03:59.274 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:59.274 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:59.274 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:03:59.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:03:59.274 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:03:59.275 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:03:59.275 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:03:59.275 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:03:59.275 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:03:59.277 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844239.051805_208.1 target=1_1631844238.977716_191.1 2021-09-17T02:04:04.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:04.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:04.280 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:04.330 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:04.330 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:04.386 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:04.387 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:04.387 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:04.388 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:04.388 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:04.388 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:04.389 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:04.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:04.390 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:04.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:04.390 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:04.391 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:04.391 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:04.392 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:04.392 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:04.392 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:04.393 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:04.393 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:04.394 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:04.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:04.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:04.395 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:04.395 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:04.396 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:04.396 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844239.051805_208.1", 2021-09-17T02:04:04.396 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:04.397 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:04.397 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:04.398 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:59.051805Z", 2021-09-17T02:04:04.398 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:04.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:04.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:04.400 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:04.400 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:04.400 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:04.401 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:04.401 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:04.401 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:04.402 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:04.402 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:04.403 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:04.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:04.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:04.404 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:04.404 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:04.404 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:04.404 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:04.404 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:04.405 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:04.405 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:04.405 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:04.405 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:04.406 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.406 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.406 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:04.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:04.406 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:04.407 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:04.407 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:04.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.407 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:04.409 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:04:04.409 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:04:04.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:04.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:04.411 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:04.461 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:04.461 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:04.517 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:04:04.517 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:04:04.518 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:04:04.518 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:04:04.518 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:04:04.518 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:04.519 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:04:04.519 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:04.519 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:04:04.519 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:04.520 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:04:04.520 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:04:04.520 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:04:04.520 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:04.520 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:04:04.521 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:04:04.521 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:04:04.521 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:04.521 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:04:04.522 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:04.522 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:04.522 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:04.522 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:04:04.523 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:04:04.523 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:04:04.523 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844239.051805_208.1", 2021-09-17T02:04:04.523 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:04:04.523 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:04:04.524 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:04:04.524 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:03:59.051805Z", 2021-09-17T02:04:04.524 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:04:04.524 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:04.525 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:04.525 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:04.525 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:04:04.525 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:04:04.525 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:04:04.526 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:04:04.526 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:04:04.526 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:04:04.526 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:04:04.526 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:04:04.527 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:04:04.527 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:04.527 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:04.527 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:04.527 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:04:04.528 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:04:04.528 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:04:04.528 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:04:04.528 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:04:04.529 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:04:04.529 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:04:04.529 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:04:04.529 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:04:04.529 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:04.530 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:04.530 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:04:04.530 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:04.530 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:04:04.531 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:04:04.531 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:04:04.531 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:04.531 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:04:04.532 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:04:04.533 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:04:04.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:04.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:04.534 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:04.580 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:04.580 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:04.628 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:04.629 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:04.629 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:04.629 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:04.629 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:04.630 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:04.630 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:04.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:04.630 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:04.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:04.631 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:04.631 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:04.631 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:04.631 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:04.632 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:04.632 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:04.632 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:04.632 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:04.633 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:04.633 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:04.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:04.633 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:04.634 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:04.634 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:04.634 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844239.051805_208.1", 2021-09-17T02:04:04.634 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:04.635 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:04.635 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:04.636 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:03:59.051805Z", 2021-09-17T02:04:04.636 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:04.637 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:04.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:04.638 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:04.638 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:04.639 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:04.639 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:04.640 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:04.640 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:04.640 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:04.641 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:04.641 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:04.642 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:04.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:04.643 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:04.643 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:04.644 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:04.644 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:04.645 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:04.645 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:04.646 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:04.646 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:04.647 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:04.647 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.648 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.648 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:04.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:04.649 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:04.649 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:04.650 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:04.650 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:04.651 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:04.652 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:04:04.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:04.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:04.822 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:04.874 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:04.875 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:04.893 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:04.892+0000 7f548775d380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:04.893 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:04.892+0000 7f548775d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:04.893 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:04.892+0000 7f548775d380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:04.894 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:04.892+0000 7f548775d380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:04.918 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:04.918+0000 7f548775d380 1 bucket sync caught up with source: 2021-09-17T02:04:04.919 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.43.11, , ] 2021-09-17T02:04:04.919 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000003.43.11, , ] 2021-09-17T02:04:04.920 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:04.918+0000 7f548775d380 0 bucket checkpoint complete 2021-09-17T02:04:04.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:04.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:04.934 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:04.988 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:04.988 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:05.009 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.007+0000 7f3d1ffeb380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:05.009 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.007+0000 7f3d1ffeb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:05.009 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.007+0000 7f3d1ffeb380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:05.009 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.007+0000 7f3d1ffeb380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:05.035 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.034+0000 7f3d1ffeb380 1 bucket sync caught up with source: 2021-09-17T02:04:05.036 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000003.43.11, , ] 2021-09-17T02:04:05.036 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000003.43.11, , ] 2021-09-17T02:04:05.036 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.034+0000 7f3d1ffeb380 0 bucket checkpoint complete 2021-09-17T02:04:05.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:05.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:05.053 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:05.105 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:05.105 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:05.126 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.125+0000 7ff6aa74e380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.125+0000 7ff6aa74e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.125+0000 7ff6aa74e380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:04:05.127 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.125+0000 7ff6aa74e380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:05.151 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.150+0000 7ff6aa74e380 1 bucket sync caught up with source: 2021-09-17T02:04:05.152 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.43.11, , ] 2021-09-17T02:04:05.152 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000003.43.11, , ] 2021-09-17T02:04:05.153 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.151+0000 7ff6aa74e380 0 bucket checkpoint complete 2021-09-17T02:04:05.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:05.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:05.165 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:05.216 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:05.217 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:05.236 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.234+0000 7fad16b6a380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:05.237 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.234+0000 7fad16b6a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:05.237 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.234+0000 7fad16b6a380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:05.238 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.234+0000 7fad16b6a380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:05.262 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.260+0000 7fad16b6a380 1 bucket sync caught up with source: 2021-09-17T02:04:05.262 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000002.354.5, , ] 2021-09-17T02:04:05.263 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000002.354.5, , ] 2021-09-17T02:04:05.263 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.260+0000 7fad16b6a380 0 bucket checkpoint complete 2021-09-17T02:04:05.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:05.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:05.281 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:05.336 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:05.337 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:05.355 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.353+0000 7f41bac6a380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:05.355 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.353+0000 7f41bac6a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:05.356 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.353+0000 7f41bac6a380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:05.356 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.353+0000 7f41bac6a380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:05.380 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.379+0000 7f41bac6a380 1 bucket sync caught up with source: 2021-09-17T02:04:05.381 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000002.354.5, , ] 2021-09-17T02:04:05.381 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000002.354.5, , ] 2021-09-17T02:04:05.381 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.379+0000 7f41bac6a380 0 bucket checkpoint complete 2021-09-17T02:04:05.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:05.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:05.395 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:05.448 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:05.448 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:05.468 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.466+0000 7fa72c4a1380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:05.468 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.466+0000 7fa72c4a1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:05.468 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.466+0000 7fa72c4a1380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:05.469 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.466+0000 7fa72c4a1380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:05.498 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.496+0000 7fa72c4a1380 1 bucket sync caught up with source: 2021-09-17T02:04:05.498 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000002.382.5, , ] 2021-09-17T02:04:05.499 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000002.382.5, , ] 2021-09-17T02:04:05.499 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:05.496+0000 7fa72c4a1380 0 bucket checkpoint complete 2021-09-17T02:04:05.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:05.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:05.514 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:05.565 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:05.584 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.583+0000 7f5b17b05380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:05.584 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.583+0000 7f5b17b05380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:05.585 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.583+0000 7f5b17b05380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:05.585 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.583+0000 7f5b17b05380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:05.609 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.609+0000 7f5b17b05380 1 bucket sync caught up with source: 2021-09-17T02:04:05.610 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000002.382.5, , ] 2021-09-17T02:04:05.610 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000002.382.5, , ] 2021-09-17T02:04:05.610 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:05.609+0000 7f5b17b05380 0 bucket checkpoint complete 2021-09-17T02:04:05.622 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-21 zones={test-zone2, test-zone1} 2021-09-17T02:04:05.628 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:05.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:05.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:05.677 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:05.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:05.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:05.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:05.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:05.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:05.797 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-21 zones={test-zone2, test-zone1} 2021-09-17T02:04:05.798 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-21 zones={test-zone3, test-zone1} 2021-09-17T02:04:05.802 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:05.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:05.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:05.848 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:05.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:05.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:05.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:05.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:05.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:05.974 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-21 zones={test-zone3, test-zone1} 2021-09-17T02:04:05.975 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-21 zones={test-zone3, test-zone2} 2021-09-17T02:04:05.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:06.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:06.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:06.024 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:06.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:06.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:06.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:06.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:06.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:06.150 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-21 zones={test-zone3, test-zone2} 2021-09-17T02:04:06.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.152 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:06.205 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.223+0000 7f10e988f380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:06.224 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.223+0000 7f10e988f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.223+0000 7f10e988f380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:06.225 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.223+0000 7f10e988f380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:06.249 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.249+0000 7f10e988f380 1 bucket sync caught up with source: 2021-09-17T02:04:06.249 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.21.11, , ] 2021-09-17T02:04:06.250 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000003.21.11, , ] 2021-09-17T02:04:06.250 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.249+0000 7f10e988f380 0 bucket checkpoint complete 2021-09-17T02:04:06.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.263 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:06.316 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:06.317 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:06.335 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.334+0000 7fabbecd4380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:06.336 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.334+0000 7fabbecd4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:06.336 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.334+0000 7fabbecd4380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:06.336 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.334+0000 7fabbecd4380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:06.360 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.359+0000 7fabbecd4380 1 bucket sync caught up with source: 2021-09-17T02:04:06.361 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000003.21.11, , ] 2021-09-17T02:04:06.361 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000003.21.11, , ] 2021-09-17T02:04:06.361 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.359+0000 7fabbecd4380 0 bucket checkpoint complete 2021-09-17T02:04:06.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.374 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:06.425 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:06.425 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:06.443 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.442+0000 7fe02e09a380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:06.443 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.442+0000 7fe02e09a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:06.444 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.442+0000 7fe02e09a380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:04:06.444 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.442+0000 7fe02e09a380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:06.467 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.467+0000 7fe02e09a380 1 bucket sync caught up with source: 2021-09-17T02:04:06.468 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.21.11, , ] 2021-09-17T02:04:06.468 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000003.21.11, , ] 2021-09-17T02:04:06.469 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.467+0000 7fe02e09a380 0 bucket checkpoint complete 2021-09-17T02:04:06.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.481 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:06.531 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:06.531 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:06.549 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.548+0000 7f8419d50380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:06.550 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.548+0000 7f8419d50380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:06.550 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.548+0000 7f8419d50380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:06.550 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.548+0000 7f8419d50380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:06.575 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.573+0000 7f8419d50380 1 bucket sync caught up with source: 2021-09-17T02:04:06.575 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000003.679.11, , ] 2021-09-17T02:04:06.575 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000003.679.11, , ] 2021-09-17T02:04:06.576 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.573+0000 7f8419d50380 0 bucket checkpoint complete 2021-09-17T02:04:06.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.590 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:06.642 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:06.642 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:06.660 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.659+0000 7fe8c4968380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:06.660 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.659+0000 7fe8c4968380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:06.661 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.659+0000 7fe8c4968380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:06.661 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.659+0000 7fe8c4968380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:06.684 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.683+0000 7fe8c4968380 1 bucket sync caught up with source: 2021-09-17T02:04:06.685 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000003.679.11, , ] 2021-09-17T02:04:06.685 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000003.679.11, , ] 2021-09-17T02:04:06.685 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.683+0000 7fe8c4968380 0 bucket checkpoint complete 2021-09-17T02:04:06.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.698 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:06.748 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:06.749 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:06.767 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.766+0000 7fcbe3648380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:06.767 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.766+0000 7fcbe3648380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:06.768 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.766+0000 7fcbe3648380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:06.769 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.766+0000 7fcbe3648380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:06.792 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.791+0000 7fcbe3648380 1 bucket sync caught up with source: 2021-09-17T02:04:06.792 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000003.653.11, , ] 2021-09-17T02:04:06.793 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000003.653.11, , ] 2021-09-17T02:04:06.793 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:06.791+0000 7fcbe3648380 0 bucket checkpoint complete 2021-09-17T02:04:06.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:06.806 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:06.858 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:06.859 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.876+0000 7f94fbaf4380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:06.877 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.876+0000 7f94fbaf4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.876+0000 7f94fbaf4380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:06.878 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.876+0000 7f94fbaf4380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:06.902 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.901+0000 7f94fbaf4380 1 bucket sync caught up with source: 2021-09-17T02:04:06.902 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.653.11, , ] 2021-09-17T02:04:06.903 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000003.653.11, , ] 2021-09-17T02:04:06.903 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:06.901+0000 7f94fbaf4380 0 bucket checkpoint complete 2021-09-17T02:04:06.915 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-22 zones={test-zone2, test-zone1} 2021-09-17T02:04:06.920 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:06.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:06.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:06.968 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:06.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:06.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:07.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:07.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:07.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:07.085 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-22 zones={test-zone2, test-zone1} 2021-09-17T02:04:07.086 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-22 zones={test-zone3, test-zone1} 2021-09-17T02:04:07.091 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:07.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:07.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:07.136 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:07.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:07.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:07.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:07.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:07.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:07.302 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-22 zones={test-zone3, test-zone1} 2021-09-17T02:04:07.303 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-22 zones={test-zone3, test-zone2} 2021-09-17T02:04:07.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:07.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:07.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:07.356 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:07.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:07.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:07.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:07.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:07.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:07.481 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-22 zones={test-zone3, test-zone2} 2021-09-17T02:04:07.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:07.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:07.483 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:07.535 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:07.535 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:07.555 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.554+0000 7f10d477f380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:07.555 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.554+0000 7f10d477f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:07.555 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.554+0000 7f10d477f380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:07.555 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.554+0000 7f10d477f380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:07.579 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.579+0000 7f10d477f380 1 bucket sync caught up with source: 2021-09-17T02:04:07.580 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000002.9.5, , ] 2021-09-17T02:04:07.580 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000002.9.5, , ] 2021-09-17T02:04:07.580 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.579+0000 7f10d477f380 0 bucket checkpoint complete 2021-09-17T02:04:07.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:07.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:07.593 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:07.651 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:07.651 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:07.673 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.671+0000 7f1d99b76380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:07.673 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.671+0000 7f1d99b76380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:07.673 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.671+0000 7f1d99b76380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:07.674 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.671+0000 7f1d99b76380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:07.700 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.699+0000 7f1d99b76380 1 bucket sync caught up with source: 2021-09-17T02:04:07.700 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000002.9.5, , ] 2021-09-17T02:04:07.701 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000002.9.5, , ] 2021-09-17T02:04:07.701 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.699+0000 7f1d99b76380 0 bucket checkpoint complete 2021-09-17T02:04:07.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:07.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:07.715 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:07.766 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:07.767 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:07.785 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.784+0000 7fdb998e4380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:07.785 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.784+0000 7fdb998e4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:07.785 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.784+0000 7fdb998e4380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:04:07.786 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.784+0000 7fdb998e4380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:07.809 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.808+0000 7fdb998e4380 1 bucket sync caught up with source: 2021-09-17T02:04:07.809 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000002.9.5, , ] 2021-09-17T02:04:07.809 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000002.9.5, , ] 2021-09-17T02:04:07.810 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:07.808+0000 7fdb998e4380 0 bucket checkpoint complete 2021-09-17T02:04:07.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:07.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:07.822 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:07.874 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:07.874 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:07.892 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.891+0000 7f5e5d442380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:07.893 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.891+0000 7f5e5d442380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:07.893 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.891+0000 7f5e5d442380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:07.893 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.891+0000 7f5e5d442380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:07.917 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.916+0000 7f5e5d442380 1 bucket sync caught up with source: 2021-09-17T02:04:07.918 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000002.133.5, , ] 2021-09-17T02:04:07.918 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000002.133.5, , ] 2021-09-17T02:04:07.918 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.916+0000 7f5e5d442380 0 bucket checkpoint complete 2021-09-17T02:04:07.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:07.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:07.931 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:07.981 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:07.982 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:08.000 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.999+0000 7f3f7f33a380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:08.000 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.999+0000 7f3f7f33a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:08.001 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.999+0000 7f3f7f33a380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:08.001 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:07.999+0000 7f3f7f33a380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:08.024 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:08.023+0000 7f3f7f33a380 1 bucket sync caught up with source: 2021-09-17T02:04:08.025 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000002.133.5, , ] 2021-09-17T02:04:08.025 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000002.133.5, , ] 2021-09-17T02:04:08.025 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:08.023+0000 7f3f7f33a380 0 bucket checkpoint complete 2021-09-17T02:04:08.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:08.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:08.038 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:08.088 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:08.089 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:08.109 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:08.108+0000 7f358a732380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:08.110 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:08.108+0000 7f358a732380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:08.110 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:08.108+0000 7f358a732380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:08.110 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:08.108+0000 7f358a732380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:08.134 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:08.132+0000 7f358a732380 1 bucket sync caught up with source: 2021-09-17T02:04:08.134 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000003.188.11, , ] 2021-09-17T02:04:08.134 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000003.188.11, , ] 2021-09-17T02:04:08.135 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:08.132+0000 7f358a732380 0 bucket checkpoint complete 2021-09-17T02:04:08.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:08.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:08.148 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:08.201 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:08.201 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:08.220 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:08.219+0000 7f6604af4380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:08.220 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:08.219+0000 7f6604af4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:08.220 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:08.219+0000 7f6604af4380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:08.221 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:08.219+0000 7f6604af4380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:08.245 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:08.245+0000 7f6604af4380 1 bucket sync caught up with source: 2021-09-17T02:04:08.246 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.188.11, , ] 2021-09-17T02:04:08.246 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000003.188.11, , ] 2021-09-17T02:04:08.246 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:08.245+0000 7f6604af4380 0 bucket checkpoint complete 2021-09-17T02:04:08.258 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-23 zones={test-zone2, test-zone1} 2021-09-17T02:04:08.263 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:08.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.312 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:08.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:08.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:08.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:08.437 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-23 zones={test-zone2, test-zone1} 2021-09-17T02:04:08.438 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-23 zones={test-zone3, test-zone1} 2021-09-17T02:04:08.442 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:08.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.492 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:08.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:08.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:08.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:08.609 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-23 zones={test-zone3, test-zone1} 2021-09-17T02:04:08.610 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-23 zones={test-zone3, test-zone2} 2021-09-17T02:04:08.614 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:08.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.660 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:08.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:08.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:08.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:08.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:08.777 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-23 zones={test-zone3, test-zone2} 2021-09-17T02:04:08.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-09-17T02:04:08.779 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-24 2021-09-17T02:04:08.792 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-25 2021-09-17T02:04:08.817 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-26 2021-09-17T02:04:08.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:08.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:08.881 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:08.930 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:08.930 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:08.971 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:04:08.983 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:04:08.983 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:08.983 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:08.984 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:04:08.984 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:08.984 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:08.985 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844248.877336_232.1", 2021-09-17T02:04:08.985 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:04:08.877336Z" 2021-09-17T02:04:08.985 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:08.985 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:08.985 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:04:08.986 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:04:08.986 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:08.986 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:08.986 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:04:08.987 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:04:08.987 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:08.987 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:04:08.988 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:04:08.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:08.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:08.990 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:09.036 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:09.037 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:09.088 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:09.088 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:09.089 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:09.089 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:09.089 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:09.089 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:09.089 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:09.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:09.090 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:09.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:09.090 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:09.091 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:09.091 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:09.091 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:09.091 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:09.091 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:09.092 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:09.092 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:09.092 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:09.092 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:09.093 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:09.093 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:09.093 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:09.093 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:09.093 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:09.094 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844248.784917_209.1", 2021-09-17T02:04:09.094 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:09.094 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:09.094 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:09.095 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:08.784917Z", 2021-09-17T02:04:09.095 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:09.095 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:09.095 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:09.095 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:09.096 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:09.096 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:09.096 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:09.096 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:09.097 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:09.097 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:09.097 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:09.097 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:09.097 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:09.098 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:09.098 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:09.098 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:09.098 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:09.098 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:09.099 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:09.099 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:09.099 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:09.099 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:09.100 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:09.100 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:09.100 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:09.100 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:09.100 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:09.101 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:09.101 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:09.101 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:09.101 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:09.101 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:09.102 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:09.102 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:09.103 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844248.877336_232.1 target=1_1631844248.784917_209.1 2021-09-17T02:04:14.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:14.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:14.107 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:14.159 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:14.160 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:14.280 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:14.280 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:14.280 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:14.281 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:14.281 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:14.281 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:14.281 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:14.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:14.282 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:14.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:14.282 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:14.282 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:14.283 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:14.283 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:14.283 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:14.283 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:14.284 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:14.284 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:14.284 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:14.284 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:14.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:14.285 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:14.285 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:14.285 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844248.877336_232.1", 2021-09-17T02:04:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:14.286 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:08.877336Z", 2021-09-17T02:04:14.287 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:14.287 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:14.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:14.288 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:14.288 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:14.288 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:14.288 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:14.288 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:14.289 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:14.289 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:14.289 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:14.289 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:14.289 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:14.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:14.290 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:14.290 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:14.291 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:14.291 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:14.291 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:14.291 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:14.291 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:14.292 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:14.292 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:14.292 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.292 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.292 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:14.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:14.293 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:14.293 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:14.293 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:14.294 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.294 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:14.295 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T02:04:14.296 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T02:04:14.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:14.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:14.297 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:14.347 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:14.347 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:14.401 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:04:14.402 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": { 2021-09-17T02:04:14.402 INFO:teuthology.orchestra.run.smithi013.stdout: "info": { 2021-09-17T02:04:14.402 INFO:teuthology.orchestra.run.smithi013.stdout: "status": "sync", 2021-09-17T02:04:14.402 INFO:teuthology.orchestra.run.smithi013.stdout: "num_shards": 4, 2021-09-17T02:04:14.403 INFO:teuthology.orchestra.run.smithi013.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:14.403 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:04:14.403 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:14.403 INFO:teuthology.orchestra.run.smithi013.stdout: "markers": [ 2021-09-17T02:04:14.404 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:14.404 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 0, 2021-09-17T02:04:14.404 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:04:14.404 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:04:14.405 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:14.405 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:04:14.405 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 1, 2021-09-17T02:04:14.405 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:04:14.405 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:14.406 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:04:14.406 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:14.406 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:14.406 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:14.407 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 1, 2021-09-17T02:04:14.407 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:04:14.407 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:04:14.407 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844248.877336_232.1", 2021-09-17T02:04:14.408 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:04:14.408 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:04:14.408 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:04:14.408 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "2021-09-17T02:04:08.877336Z", 2021-09-17T02:04:14.409 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:04:14.409 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:14.409 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:14.409 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:14.409 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 2, 2021-09-17T02:04:14.410 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:04:14.410 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:04:14.410 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:04:14.410 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:04:14.411 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:04:14.411 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:04:14.411 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:04:14.411 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:04:14.412 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:14.412 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:14.412 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:14.412 INFO:teuthology.orchestra.run.smithi013.stdout: "key": 3, 2021-09-17T02:04:14.412 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:04:14.413 INFO:teuthology.orchestra.run.smithi013.stdout: "state": 1, 2021-09-17T02:04:14.413 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:04:14.413 INFO:teuthology.orchestra.run.smithi013.stdout: "next_step_marker": "", 2021-09-17T02:04:14.413 INFO:teuthology.orchestra.run.smithi013.stdout: "total_entries": 0, 2021-09-17T02:04:14.414 INFO:teuthology.orchestra.run.smithi013.stdout: "pos": 0, 2021-09-17T02:04:14.414 INFO:teuthology.orchestra.run.smithi013.stdout: "timestamp": "0.000000", 2021-09-17T02:04:14.414 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:04:14.414 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:14.415 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:14.415 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:04:14.415 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:14.415 INFO:teuthology.orchestra.run.smithi013.stdout: "full_sync": { 2021-09-17T02:04:14.415 INFO:teuthology.orchestra.run.smithi013.stdout: "total": 1, 2021-09-17T02:04:14.416 INFO:teuthology.orchestra.run.smithi013.stdout: "complete": 1 2021-09-17T02:04:14.416 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:14.416 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:04:14.417 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T02:04:14.418 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T02:04:14.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:14.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:14.419 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:14.467 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:14.468 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:14.519 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:14.519 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:14.519 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:14.520 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:14.520 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:14.520 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:14.520 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:14.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:14.521 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:14.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:14.521 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:14.522 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:14.522 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:14.522 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:14.522 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:14.523 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:14.523 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:14.523 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:14.523 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:14.523 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:14.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:14.524 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:14.524 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:14.524 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:14.525 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844248.877336_232.1", 2021-09-17T02:04:14.525 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:14.525 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:14.525 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:14.525 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:08.877336Z", 2021-09-17T02:04:14.526 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:14.526 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:14.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:14.527 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:14.527 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:14.527 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:14.527 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:14.527 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:14.528 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:14.528 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:14.528 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:14.528 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:14.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:14.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:14.529 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:14.529 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:14.530 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:14.530 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:14.530 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:14.530 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:14.531 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:14.531 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:14.531 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:14.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.532 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:14.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:14.533 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:14.533 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:14.533 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:14.533 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:14.533 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:14.535 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T02:04:14.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:14.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:14.723 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:14.775 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:14.775 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:14.793 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:14.793+0000 7f2134e5e380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:14.794 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:14.793+0000 7f2134e5e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:14.794 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:14.793+0000 7f2134e5e380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:14.794 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:14.793+0000 7f2134e5e380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:14.823 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:14.822+0000 7f2134e5e380 1 bucket sync caught up with source: 2021-09-17T02:04:14.823 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.55.6, , ] 2021-09-17T02:04:14.823 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000003.55.6, , ] 2021-09-17T02:04:14.824 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:14.822+0000 7f2134e5e380 0 bucket checkpoint complete 2021-09-17T02:04:14.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:14.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:14.837 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:14.894 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:14.894 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:14.916 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:14.914+0000 7eff029d9380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:14.916 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:14.914+0000 7eff029d9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:14.917 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:14.914+0000 7eff029d9380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:14.917 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:14.914+0000 7eff029d9380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:14.946 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:14.945+0000 7eff029d9380 1 bucket sync caught up with source: 2021-09-17T02:04:14.947 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000003.55.6, , ] 2021-09-17T02:04:14.947 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000003.55.6, , ] 2021-09-17T02:04:14.947 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:14.945+0000 7eff029d9380 0 bucket checkpoint complete 2021-09-17T02:04:14.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:14.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:14.964 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:15.016 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:15.017 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.034+0000 7f348e175380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:15.035 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.034+0000 7f348e175380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:15.036 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.034+0000 7f348e175380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:04:15.036 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.034+0000 7f348e175380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:15.060 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.059+0000 7f348e175380 1 bucket sync caught up with source: 2021-09-17T02:04:15.060 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.55.6, , ] 2021-09-17T02:04:15.061 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000003.55.6, , ] 2021-09-17T02:04:15.061 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.059+0000 7f348e175380 0 bucket checkpoint complete 2021-09-17T02:04:15.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.075 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:15.126 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:15.127 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:15.145 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.144+0000 7fcd1188f380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:15.145 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.144+0000 7fcd1188f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:15.145 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.144+0000 7fcd1188f380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:15.146 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.144+0000 7fcd1188f380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:15.171 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.169+0000 7fcd1188f380 1 bucket sync caught up with source: 2021-09-17T02:04:15.171 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.365.12, , ] 2021-09-17T02:04:15.171 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.365.12, , ] 2021-09-17T02:04:15.171 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.169+0000 7fcd1188f380 0 bucket checkpoint complete 2021-09-17T02:04:15.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.185 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:15.237 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:15.237 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:15.255 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.254+0000 7f64d6043380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:15.256 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.254+0000 7f64d6043380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:15.256 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.254+0000 7f64d6043380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:15.257 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.254+0000 7f64d6043380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:15.280 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.279+0000 7f64d6043380 1 bucket sync caught up with source: 2021-09-17T02:04:15.281 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.365.12, , ] 2021-09-17T02:04:15.281 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.365.12, , ] 2021-09-17T02:04:15.281 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.279+0000 7f64d6043380 0 bucket checkpoint complete 2021-09-17T02:04:15.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.294 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:15.344 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:15.344 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:15.363 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.361+0000 7fe14a7df380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:15.363 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.361+0000 7fe14a7df380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:15.364 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.361+0000 7fe14a7df380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:15.365 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.361+0000 7fe14a7df380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:15.388 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.386+0000 7fe14a7df380 1 bucket sync caught up with source: 2021-09-17T02:04:15.388 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.391.12, , ] 2021-09-17T02:04:15.388 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.391.12, , ] 2021-09-17T02:04:15.389 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:15.386+0000 7fe14a7df380 0 bucket checkpoint complete 2021-09-17T02:04:15.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.402 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:15.474 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.473+0000 7f6609de5380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:15.474 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.473+0000 7f6609de5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:15.474 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.473+0000 7f6609de5380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:15.475 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.473+0000 7f6609de5380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:15.499 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.498+0000 7f6609de5380 1 bucket sync caught up with source: 2021-09-17T02:04:15.499 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000001.391.12, , ] 2021-09-17T02:04:15.499 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000001.391.12, , ] 2021-09-17T02:04:15.500 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.498+0000 7f6609de5380 0 bucket checkpoint complete 2021-09-17T02:04:15.511 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-24 zones={test-zone2, test-zone1} 2021-09-17T02:04:15.517 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:15.567 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:15.568 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:15.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:15.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:15.625 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-24 zones={test-zone2, test-zone1} 2021-09-17T02:04:15.625 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-24 zones={test-zone3, test-zone1} 2021-09-17T02:04:15.630 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:15.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:15.676 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:15.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:15.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:15.733 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-24 zones={test-zone3, test-zone1} 2021-09-17T02:04:15.734 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-24 zones={test-zone3, test-zone2} 2021-09-17T02:04:15.738 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:15.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:15.783 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:15.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:15.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:15.837 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-24 zones={test-zone3, test-zone2} 2021-09-17T02:04:15.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.838 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:15.889 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:15.908 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.907+0000 7fb2066ae380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:15.908 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.907+0000 7fb2066ae380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:15.908 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.907+0000 7fb2066ae380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:15.909 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.907+0000 7fb2066ae380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:15.933 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.932+0000 7fb2066ae380 1 bucket sync caught up with source: 2021-09-17T02:04:15.933 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000001.16.12, , ] 2021-09-17T02:04:15.934 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000001.16.12, , ] 2021-09-17T02:04:15.934 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:15.932+0000 7fb2066ae380 0 bucket checkpoint complete 2021-09-17T02:04:15.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:15.947 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:16.000 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:16.001 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:16.021 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:16.020+0000 7f4432e53380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:16.022 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:16.020+0000 7f4432e53380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:16.022 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:16.020+0000 7f4432e53380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:16.022 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:16.020+0000 7f4432e53380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:16.047 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:16.046+0000 7f4432e53380 1 bucket sync caught up with source: 2021-09-17T02:04:16.048 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.16.12, , ] 2021-09-17T02:04:16.048 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.16.12, , ] 2021-09-17T02:04:16.048 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:16.046+0000 7f4432e53380 0 bucket checkpoint complete 2021-09-17T02:04:16.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:16.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:16.062 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:16.113 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:16.130+0000 7f4d18dae380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:16.130+0000 7f4d18dae380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:16.131 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:16.130+0000 7f4d18dae380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:04:16.132 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:16.130+0000 7f4d18dae380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:16.155 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:16.155+0000 7f4d18dae380 1 bucket sync caught up with source: 2021-09-17T02:04:16.156 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000001.16.12, , ] 2021-09-17T02:04:16.156 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000001.16.12, , ] 2021-09-17T02:04:16.156 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:16.155+0000 7f4d18dae380 0 bucket checkpoint complete 2021-09-17T02:04:16.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:16.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:16.169 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:16.220 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:16.221 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:16.238 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:16.237+0000 7f8dcf4c5380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:16.239 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:16.237+0000 7f8dcf4c5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:16.239 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:16.237+0000 7f8dcf4c5380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:16.239 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:16.237+0000 7f8dcf4c5380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:17.266 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.264+0000 7f8dcf4c5380 1 bucket sync caught up with source: 2021-09-17T02:04:17.267 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000003.263.6, , ] 2021-09-17T02:04:17.267 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000003.263.6, , ] 2021-09-17T02:04:17.267 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.264+0000 7f8dcf4c5380 0 bucket checkpoint complete 2021-09-17T02:04:17.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:17.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:17.286 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:17.344 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:17.345 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:17.366 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.364+0000 7f62e1cdd380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:17.366 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.364+0000 7f62e1cdd380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:17.366 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.364+0000 7f62e1cdd380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:17.367 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.364+0000 7f62e1cdd380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:17.393 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.392+0000 7f62e1cdd380 1 bucket sync caught up with source: 2021-09-17T02:04:17.393 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000003.263.6, , ] 2021-09-17T02:04:17.394 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000003.263.6, , ] 2021-09-17T02:04:17.394 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.392+0000 7f62e1cdd380 0 bucket checkpoint complete 2021-09-17T02:04:17.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:17.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:17.408 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:17.459 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:17.459 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:17.477 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.476+0000 7f3235209380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:17.478 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.476+0000 7f3235209380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:17.478 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.476+0000 7f3235209380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:17.478 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.476+0000 7f3235209380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:17.502 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.501+0000 7f3235209380 1 bucket sync caught up with source: 2021-09-17T02:04:17.503 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.205.12, , ] 2021-09-17T02:04:17.503 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.205.12, , ] 2021-09-17T02:04:17.503 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:17.501+0000 7f3235209380 0 bucket checkpoint complete 2021-09-17T02:04:17.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:17.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:17.517 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:17.574 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:17.575 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:17.595+0000 7fdb78f68380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:17.595+0000 7fdb78f68380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:17.596 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:17.595+0000 7fdb78f68380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:17.597 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:17.595+0000 7fdb78f68380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:17.624+0000 7fdb78f68380 1 bucket sync caught up with source: 2021-09-17T02:04:17.625 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000001.205.12, , ] 2021-09-17T02:04:17.625 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000001.205.12, , ] 2021-09-17T02:04:17.625 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:17.624+0000 7fdb78f68380 0 bucket checkpoint complete 2021-09-17T02:04:17.638 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-25 zones={test-zone2, test-zone1} 2021-09-17T02:04:17.643 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:17.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:17.687 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:17.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:17.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:17.741 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-25 zones={test-zone2, test-zone1} 2021-09-17T02:04:17.742 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-25 zones={test-zone3, test-zone1} 2021-09-17T02:04:17.747 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:17.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:17.791 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:17.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:17.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:17.845 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-25 zones={test-zone3, test-zone1} 2021-09-17T02:04:17.845 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-25 zones={test-zone3, test-zone2} 2021-09-17T02:04:17.850 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:17.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:17.896 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:17.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:17.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:17.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-25 zones={test-zone3, test-zone2} 2021-09-17T02:04:17.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:17.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:17.951 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:18.026 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.025+0000 7f774c72f380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:18.027 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.025+0000 7f774c72f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:18.027 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.025+0000 7f774c72f380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:18.027 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.025+0000 7f774c72f380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:18.053 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.052+0000 7f774c72f380 1 bucket sync caught up with source: 2021-09-17T02:04:18.053 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000001.33.12, , ] 2021-09-17T02:04:18.053 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000001.33.12, , ] 2021-09-17T02:04:18.054 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.052+0000 7f774c72f380 0 bucket checkpoint complete 2021-09-17T02:04:18.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.067 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:18.122 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:18.123 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:18.142 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.140+0000 7ff6961e7380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:18.142 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.140+0000 7ff6961e7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:18.142 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.140+0000 7ff6961e7380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:18.143 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.140+0000 7ff6961e7380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:18.167 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.166+0000 7ff6961e7380 1 bucket sync caught up with source: 2021-09-17T02:04:18.167 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.33.12, , ] 2021-09-17T02:04:18.168 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.33.12, , ] 2021-09-17T02:04:18.168 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.166+0000 7ff6961e7380 0 bucket checkpoint complete 2021-09-17T02:04:18.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.181 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:18.233 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:18.234 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.252+0000 7f1be0f68380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:18.252 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.252+0000 7f1be0f68380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.252+0000 7f1be0f68380 1 Zone: test-zone4 (e8223d60-810a-44ad-a8ee-67d9caa7bcb0) 2021-09-17T02:04:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.252+0000 7f1be0f68380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:18.276 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.275+0000 7f1be0f68380 1 bucket sync caught up with source: 2021-09-17T02:04:18.276 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000001.33.12, , ] 2021-09-17T02:04:18.276 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000001.33.12, , ] 2021-09-17T02:04:18.277 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.275+0000 7f1be0f68380 0 bucket checkpoint complete 2021-09-17T02:04:18.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.289 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:18.341 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:18.342 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:18.360 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.358+0000 7f244f3f7380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:18.360 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.358+0000 7f244f3f7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:18.360 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.358+0000 7f244f3f7380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:18.361 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.358+0000 7f244f3f7380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:18.386 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.384+0000 7f244f3f7380 1 bucket sync caught up with source: 2021-09-17T02:04:18.386 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.615.12, , ] 2021-09-17T02:04:18.386 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.615.12, , ] 2021-09-17T02:04:18.387 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.384+0000 7f244f3f7380 0 bucket checkpoint complete 2021-09-17T02:04:18.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.400 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:18.450 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:18.450 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:18.469 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.467+0000 7f4afac89380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:18.469 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.467+0000 7f4afac89380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:18.469 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.467+0000 7f4afac89380 1 Zone: test-zone3 (505278ea-3c6d-4c68-8442-578099047a49) 2021-09-17T02:04:18.469 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.467+0000 7f4afac89380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:18.493 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.491+0000 7f4afac89380 1 bucket sync caught up with source: 2021-09-17T02:04:18.493 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000001.615.12, , ] 2021-09-17T02:04:18.493 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000001.615.12, , ] 2021-09-17T02:04:18.494 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.491+0000 7f4afac89380 0 bucket checkpoint complete 2021-09-17T02:04:18.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.506 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qdiasb-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:18.556 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:18.556 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:18.575 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.573+0000 7fba15356380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:18.575 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.573+0000 7fba15356380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:18.575 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.573+0000 7fba15356380 1 Zone: test-zone1 (95b0d488-6c59-4a37-b3ed-d5fa65f27678) 2021-09-17T02:04:18.575 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.573+0000 7fba15356380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:18.599 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.598+0000 7fba15356380 1 bucket sync caught up with source: 2021-09-17T02:04:18.599 INFO:teuthology.orchestra.run.smithi013.stderr: local status: [, , , , , , , , 00000000003.691.6, , ] 2021-09-17T02:04:18.600 INFO:teuthology.orchestra.run.smithi013.stderr: remote markers: [, , , , , , , , 00000000003.691.6, , ] 2021-09-17T02:04:18.600 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:04:18.598+0000 7fba15356380 0 bucket checkpoint complete 2021-09-17T02:04:18.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'qdiasb-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T02:04:18.613 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket qdiasb-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T02:04:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:18.665 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:18.683 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.683+0000 7efed495f380 1 Realm: test-realm (701b5af2-9aae-4e89-bd12-7ed067bbad31) 2021-09-17T02:04:18.684 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.683+0000 7efed495f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T02:04:18.684 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.683+0000 7efed495f380 1 Zone: test-zone2 (54e01141-a1e3-4035-9545-e42ec73c8edc) 2021-09-17T02:04:18.684 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.683+0000 7efed495f380 1 using period configuration: 3c964c05-2a9c-48af-9858-975bf7b57c1a:4 2021-09-17T02:04:18.708 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.707+0000 7efed495f380 1 bucket sync caught up with source: 2021-09-17T02:04:18.708 INFO:teuthology.orchestra.run.smithi045.stderr: local status: [, , , , , , , , 00000000003.691.6, , ] 2021-09-17T02:04:18.708 INFO:teuthology.orchestra.run.smithi045.stderr: remote markers: [, , , , , , , , 00000000003.691.6, , ] 2021-09-17T02:04:18.709 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:04:18.707+0000 7efed495f380 0 bucket checkpoint complete 2021-09-17T02:04:18.721 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-26 zones={test-zone2, test-zone1} 2021-09-17T02:04:18.726 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:18.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:18.772 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:18.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:18.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:18.824 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-26 zones={test-zone2, test-zone1} 2021-09-17T02:04:18.825 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-26 zones={test-zone3, test-zone1} 2021-09-17T02:04:18.829 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:18.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:18.875 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:18.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:18.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:18.930 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-26 zones={test-zone3, test-zone1} 2021-09-17T02:04:18.930 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qdiasb-26 zones={test-zone3, test-zone2} 2021-09-17T02:04:18.935 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T02:04:18.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:18.984 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T02:04:18.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T02:04:19.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T02:04:19.037 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=qdiasb-26 zones={test-zone3, test-zone2} 2021-09-17T02:04:19.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-09-17T02:04:19.038 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-27 2021-09-17T02:04:19.053 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-28 2021-09-17T02:04:19.077 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-29 2021-09-17T02:04:19.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2021-09-17T02:04:19.260 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-30 2021-09-17T02:04:19.274 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-31 2021-09-17T02:04:19.301 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-32 2021-09-17T02:04:19.499 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2021-09-17T02:04:19.500 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-33 2021-09-17T02:04:19.532 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-34 2021-09-17T02:04:19.633 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-35 2021-09-17T02:04:21.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2021-09-17T02:04:21.844 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-36 2021-09-17T02:04:21.863 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-37 2021-09-17T02:04:21.895 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-38 2021-09-17T02:04:21.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:21.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:21.931 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:21.977 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:21.978 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:22.019 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:04:22.032 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:04:22.032 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:22.032 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:22.033 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:04:22.033 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:22.033 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:22.033 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844261.911090_301.1", 2021-09-17T02:04:22.034 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:04:21.911090Z" 2021-09-17T02:04:22.034 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:22.034 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:22.034 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:04:22.035 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:04:22.035 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:04:22.035 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:04:22.035 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:04:22.035 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:04:22.036 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:04:22.036 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:04:22.037 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:04:22.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:22.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:22.039 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:22.088 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:22.088 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:22.143 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:22.143 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:22.144 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:22.144 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:22.144 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:22.144 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:22.145 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:22.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:22.145 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:22.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:22.145 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:22.146 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:22.146 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:22.146 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:22.146 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:22.147 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:22.147 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:22.147 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:22.147 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:22.147 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:22.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:22.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:22.148 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:22.148 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:22.149 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:22.149 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.091993_244.1", 2021-09-17T02:04:22.149 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:22.150 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:22.150 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:22.151 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.091993Z", 2021-09-17T02:04:22.151 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:22.151 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:22.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:22.152 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:22.152 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:22.152 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:22.152 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:22.153 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:22.153 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:22.153 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:22.153 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:22.154 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:22.154 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:22.154 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:22.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:22.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:22.155 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:22.155 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:22.155 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:22.155 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:22.156 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:22.156 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:22.156 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:22.156 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:22.157 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:22.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:22.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:22.157 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:22.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:22.158 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:22.158 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:22.158 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:22.159 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:22.159 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:22.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.091993_244.1 2021-09-17T02:04:27.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:27.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:27.163 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:27.234 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:27.299 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:27.300 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:27.300 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:27.300 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:27.301 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:27.301 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:27.301 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:27.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:27.301 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:27.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:27.302 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:27.302 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:27.302 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:27.303 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:27.303 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:27.303 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:27.303 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:27.304 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:27.304 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:27.304 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:27.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:27.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:27.305 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:27.305 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:27.305 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:27.305 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:04:27.306 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:27.306 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:27.306 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:27.306 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:04:27.306 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:27.307 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:27.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:27.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:27.307 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:27.308 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:27.308 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:27.308 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:27.308 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:27.308 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:27.309 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:27.309 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:27.309 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:27.309 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:27.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:27.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:27.310 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:27.310 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:27.310 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:27.311 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:27.311 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:27.311 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:27.311 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:27.311 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:27.312 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:27.312 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:27.312 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:27.312 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:27.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:27.313 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:27.313 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:27.313 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:27.313 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:27.314 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:27.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:04:32.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:32.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:32.318 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:32.362 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:32.363 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:32.412 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:32.413 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:32.413 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:32.413 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:32.414 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:32.414 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:32.414 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:32.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:32.415 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:32.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:32.415 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:32.415 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:32.415 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:32.416 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:32.417 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:32.417 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:32.417 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:32.418 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:32.418 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:32.418 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:32.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:32.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:32.419 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:32.419 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:32.419 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:32.420 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.051622_236.1", 2021-09-17T02:04:32.420 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:32.420 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:32.420 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:32.420 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.051622Z", 2021-09-17T02:04:32.421 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:32.421 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:32.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:32.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:32.422 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:32.422 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:32.422 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:32.422 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:32.422 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:32.423 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:32.423 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:32.423 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:32.423 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:32.424 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:32.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:32.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:32.424 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:32.424 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:32.425 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:32.425 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:32.425 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:32.425 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:32.425 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:32.426 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:32.426 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:32.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:32.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:32.427 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:32.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:32.427 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:32.427 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:32.428 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:32.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:32.428 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:32.429 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.051622_236.1 2021-09-17T02:04:37.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:37.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:37.433 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:37.507 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:37.508 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:37.570 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:37.571 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:37.571 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:37.571 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:37.571 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:37.572 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:37.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:37.572 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:37.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:37.572 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:37.573 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:37.573 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:37.574 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:37.574 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:37.574 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:37.574 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:37.575 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:37.575 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:37.575 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:37.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:37.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:37.576 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:37.576 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:37.576 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:37.576 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:04:37.577 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:37.577 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:37.577 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:37.577 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:04:37.578 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:37.578 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:37.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:37.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:37.578 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:37.579 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:37.579 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:37.579 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:37.580 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:37.580 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:37.580 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:37.580 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:37.580 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:37.581 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:37.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:37.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:37.581 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:37.581 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:37.582 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:37.582 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:37.582 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:37.582 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:37.583 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:37.583 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:37.583 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:37.583 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:37.583 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:37.584 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:37.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:37.584 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:37.584 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:37.585 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:37.585 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:37.585 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:37.587 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:04:42.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:42.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:42.590 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:42.634 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:42.634 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:42.683 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:42.684 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:42.684 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:42.684 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:42.684 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:42.685 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:42.685 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:42.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:42.685 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:42.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:42.686 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:42.686 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:42.686 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:42.687 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:42.687 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:42.687 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:42.687 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:42.687 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:42.688 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:42.688 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:42.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:42.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:42.689 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:42.689 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:42.689 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:42.689 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:04:42.689 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:42.690 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:42.690 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:42.690 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:04:42.690 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:42.691 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:42.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:42.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:42.691 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:42.691 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:42.692 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:42.692 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:42.692 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:42.692 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:42.693 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:42.693 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:42.693 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:42.693 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:42.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:42.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:42.694 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:42.694 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:42.694 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:42.694 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:42.695 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:42.695 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:42.695 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:42.695 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:42.696 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:42.696 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:42.696 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:42.696 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:42.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:42.697 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:42.697 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:42.697 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:42.697 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:42.698 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:42.699 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:04:46.097 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T02:04:46.095+0000 7f4365fc9700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-17T02:04:47.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:47.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:47.702 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:47.752 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:47.752 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:47.809 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:47.809 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:47.809 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:47.810 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:47.810 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:47.810 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:47.810 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:47.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:47.811 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:47.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:47.811 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:47.811 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:47.812 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:47.812 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:47.812 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:47.812 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:47.812 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:47.813 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:47.813 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:47.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:47.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:47.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:47.814 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:47.814 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:47.814 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:47.814 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:04:47.815 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:47.815 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:47.815 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:47.815 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:04:47.816 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:47.816 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:47.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:47.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:47.816 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:47.817 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:47.817 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:47.817 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:47.817 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:47.818 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:47.818 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:47.818 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:47.818 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:47.818 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:47.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:47.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:47.819 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:47.819 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:47.819 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:47.820 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:47.820 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:47.820 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:47.820 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:47.820 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:47.821 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:47.821 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:47.821 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:47.821 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:47.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:47.822 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:47.822 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:47.822 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:47.822 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:47.823 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:47.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:04:52.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:52.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:52.826 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:52.876 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:52.877 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:52.934 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:52.934 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:52.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:52.936 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:52.937 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:52.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:52.939 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:04:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:52.940 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:04:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:52.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:52.942 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:52.943 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:52.944 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:52.945 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:52.948 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:52.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.088776_242.1 2021-09-17T02:04:57.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:57.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:04:57.952 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:04:58.003 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:04:58.003 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:04:58.064 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:04:58.065 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:04:58.065 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:04:58.065 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:04:58.066 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:04:58.066 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:04:58.066 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:58.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:58.067 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:04:58.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:58.067 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:04:58.067 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:58.068 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:58.068 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:04:58.068 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:58.068 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:04:58.069 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:58.069 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:04:58.069 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:58.069 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:58.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:58.070 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:58.070 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:04:58.070 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:58.070 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:58.071 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.103309_245.1", 2021-09-17T02:04:58.071 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:58.072 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:58.073 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:58.073 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.103309Z", 2021-09-17T02:04:58.073 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:58.074 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:58.074 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:58.074 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:58.074 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:04:58.075 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:58.075 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:58.075 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:58.075 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:58.076 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:58.076 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:58.076 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:58.076 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:58.077 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:58.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:58.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:04:58.077 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:04:58.078 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:04:58.078 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:04:58.078 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:04:58.078 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:04:58.078 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:04:58.079 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:04:58.079 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:04:58.079 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:04:58.079 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:58.080 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:58.080 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:04:58.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:04:58.080 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:04:58.081 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:04:58.081 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:04:58.081 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:04:58.081 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:04:58.083 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.103309_245.1 2021-09-17T02:05:03.084 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:03.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:03.086 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:05:03.166 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:05:03.167 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:05:03.315 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:05:03.315 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:05:03.315 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:05:03.316 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:05:03.316 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:05:03.316 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:05:03.316 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:03.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:03.317 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:05:03.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:03.317 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:05:03.317 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:03.318 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:03.318 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:05:03.318 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:03.318 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:05:03.319 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:03.319 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:05:03.319 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:03.319 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:03.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:03.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:03.320 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:05:03.320 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:03.320 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:03.321 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:05:03.321 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:03.321 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:03.321 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:03.322 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:05:03.322 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:03.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:03.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:03.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:03.323 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:05:03.323 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:03.323 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:03.323 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:03.324 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:03.324 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:03.324 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:03.324 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:03.324 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:03.325 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:03.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:03.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:03.325 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:05:03.326 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:03.326 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:03.326 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:03.326 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:03.326 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:03.327 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:03.327 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:03.327 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:03.327 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:03.328 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:03.328 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:05:03.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:03.328 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:05:03.329 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:05:03.329 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:05:03.329 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:03.329 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:05:03.330 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.064665_238.1 2021-09-17T02:05:08.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:08.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:08.333 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:05:08.377 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:05:08.378 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:05:08.427 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:05:08.428 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:05:08.428 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:05:08.428 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:05:08.428 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:05:08.429 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:05:08.429 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:08.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:08.429 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:05:08.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:08.430 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:05:08.430 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:08.430 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:08.430 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:05:08.431 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:08.431 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:05:08.431 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:08.431 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:05:08.432 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:08.432 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:08.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:08.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:08.432 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:05:08.433 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:08.433 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:08.433 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:05:08.433 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:08.433 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:08.434 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:08.434 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:05:08.434 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:08.434 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:08.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:08.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:08.435 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:05:08.435 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:08.435 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:08.436 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:08.436 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:08.436 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:08.436 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:08.436 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:08.437 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:08.437 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:08.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:08.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:08.437 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:05:08.438 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:08.438 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:08.438 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:08.438 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:08.439 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:08.439 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:08.439 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:08.439 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:08.439 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:08.440 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:08.440 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:05:08.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:08.440 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:05:08.440 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:05:08.441 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:05:08.441 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:08.441 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:05:08.442 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.065659_239.1 2021-09-17T02:05:13.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:13.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:13.445 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:05:13.492 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:05:13.492 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:05:13.544 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:05:13.545 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:05:13.545 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:05:13.545 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:05:13.545 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:05:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:05:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:13.546 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:05:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:05:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:13.547 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:05:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:05:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:13.548 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:05:13.549 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:13.549 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:13.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:13.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:13.550 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:05:13.550 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:13.550 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:13.550 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:05:13.550 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:13.551 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:13.551 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:13.551 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:05:13.551 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:13.552 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:13.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:13.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:13.552 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:05:13.553 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:13.553 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:13.553 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:13.553 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:13.553 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:13.554 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:13.554 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:13.554 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:13.554 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:13.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:13.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:13.555 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:05:13.555 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:13.555 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:13.556 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:13.556 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:13.556 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:13.556 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:13.557 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:13.557 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:13.557 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:13.558 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:13.558 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:05:13.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:13.558 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:05:13.558 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:05:13.559 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:05:13.559 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:13.559 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:05:13.560 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.269640_253.1 2021-09-17T02:05:18.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:18.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:18.563 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:05:18.610 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:05:18.611 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:05:18.663 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:05:18.664 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:05:18.664 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:05:18.664 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:05:18.664 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:05:18.665 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:05:18.665 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:18.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:18.665 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:05:18.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:18.666 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:05:18.666 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:18.666 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:18.666 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:05:18.667 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:18.667 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:05:18.667 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:18.667 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:05:18.668 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:18.668 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:18.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:18.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:18.669 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:05:18.669 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:18.669 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:18.669 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:05:18.669 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:18.670 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:18.670 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:18.670 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:05:18.671 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:18.671 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:18.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:18.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:18.672 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:05:18.672 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:18.672 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:18.672 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:18.673 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:18.673 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:18.673 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:18.673 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:18.673 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:18.674 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:18.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:18.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:18.675 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:05:18.675 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:18.675 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:18.675 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:18.676 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:18.676 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:18.676 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:18.676 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:18.676 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:18.677 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:18.677 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:18.677 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:05:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:05:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:05:18.678 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:05:18.679 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:18.679 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:05:18.680 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.048044_234.1 2021-09-17T02:05:23.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:23.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:23.683 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:05:23.766 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:05:23.767 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:05:24.070 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:05:24.070 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:05:24.071 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:05:24.071 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:05:24.071 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:05:24.071 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:05:24.072 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:24.072 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:24.072 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:05:24.072 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:24.072 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:05:24.073 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:24.073 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:24.073 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:05:24.073 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:24.074 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:05:24.074 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:24.074 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:05:24.074 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:24.075 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:24.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:24.075 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:24.075 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:05:24.076 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:24.076 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:24.076 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:05:24.076 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:24.077 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:24.077 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:24.077 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:05:24.077 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:24.078 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:24.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:24.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:24.078 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:05:24.078 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:24.079 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:24.079 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:24.079 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:24.079 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:24.080 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:24.080 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:24.080 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:24.080 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:24.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:24.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:24.081 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:05:24.081 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:24.081 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:24.081 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:24.082 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:24.082 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:24.082 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:24.082 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:24.083 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:24.083 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:24.083 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:24.083 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:05:24.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:24.084 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:05:24.084 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:05:24.084 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:05:24.084 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:24.085 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:05:24.085 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.089761_243.1 2021-09-17T02:05:29.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:29.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:29.088 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:05:29.131 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:05:29.131 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:05:29.182 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:05:29.183 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:05:29.183 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:05:29.183 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:05:29.183 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:05:29.184 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:05:29.184 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:29.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:29.184 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:05:29.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:29.185 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:05:29.185 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:29.185 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:29.185 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:05:29.186 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:29.186 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:05:29.186 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:29.186 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:05:29.186 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:29.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:29.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:29.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:29.187 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:05:29.188 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:29.188 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:29.188 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.387102_265.1", 2021-09-17T02:05:29.188 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:29.188 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:29.189 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:29.189 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.387102Z", 2021-09-17T02:05:29.189 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:29.189 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:29.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:29.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:29.190 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:05:29.190 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:29.190 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:29.190 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:29.191 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:29.191 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:29.191 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:29.191 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:29.191 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:29.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:29.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:29.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:29.192 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:05:29.193 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:29.193 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:29.193 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:29.193 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:29.193 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:29.194 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:29.194 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:29.194 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:29.194 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:29.194 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:29.195 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:05:29.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:29.195 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:05:29.195 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:05:29.195 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:05:29.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:29.196 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:05:29.197 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.387102_265.1 2021-09-17T02:05:34.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:34.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:34.200 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:05:34.251 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:05:34.251 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:05:34.308 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:05:34.309 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:05:34.309 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:05:34.309 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:05:34.309 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:05:34.310 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:05:34.310 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:34.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:34.310 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:05:34.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:34.311 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:05:34.311 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:34.311 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:34.311 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:05:34.312 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:34.312 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:05:34.312 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:34.312 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:05:34.313 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:34.313 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:34.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:34.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:34.313 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:05:34.314 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:34.314 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:34.314 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:05:34.314 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:34.315 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:34.315 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:34.315 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:05:34.316 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:34.316 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:34.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:34.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:34.317 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:05:34.317 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:34.317 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:34.317 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:34.318 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:34.318 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:34.318 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:34.318 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:34.319 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:34.319 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:34.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:34.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:34.320 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:05:34.320 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:34.320 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:34.320 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:34.321 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:34.321 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:34.321 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:34.321 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:34.322 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:34.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:34.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:34.323 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:05:34.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:34.323 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:05:34.323 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:05:34.323 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:05:34.324 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:34.324 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:05:34.325 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.269640_253.1 2021-09-17T02:05:39.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:39.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:39.329 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:05:39.391 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:05:39.392 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:05:39.638 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:05:39.638 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:05:39.639 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:05:39.639 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:05:39.639 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:05:39.639 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:05:39.640 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:39.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:39.640 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:05:39.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:39.640 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:05:39.641 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:39.641 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:39.641 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:05:39.641 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:39.642 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:05:39.642 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:39.642 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:05:39.642 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:39.643 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:39.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:39.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:39.643 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:05:39.643 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:39.644 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:39.644 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:05:39.644 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:39.644 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:39.644 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:39.645 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:05:39.645 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:39.645 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:39.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:39.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:39.646 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:05:39.646 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:39.646 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:39.646 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:39.647 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:39.647 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:39.647 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:39.647 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:39.648 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:39.648 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:39.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:39.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:39.648 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:05:39.649 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:39.649 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:39.649 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:39.649 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:39.649 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:39.650 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:39.650 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:39.650 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:39.650 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:39.650 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:39.651 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:05:39.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:39.651 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:05:39.651 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:05:39.652 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:05:39.652 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:39.652 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:05:39.653 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.269640_253.1 2021-09-17T02:05:44.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:44.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:44.656 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:05:44.753 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:05:44.753 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:05:45.190 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:05:45.190 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:05:45.191 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:05:45.191 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:05:45.191 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:05:45.191 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:05:45.192 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:45.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:45.192 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:05:45.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:45.193 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:05:45.193 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:45.193 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:45.193 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:05:45.193 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:45.194 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:05:45.194 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:45.194 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:05:45.194 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:45.195 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:45.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:45.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:45.195 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:05:45.196 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:45.196 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:45.196 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:05:45.196 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:45.196 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:45.197 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:45.197 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:05:45.197 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:45.197 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:45.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:45.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:45.198 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:05:45.198 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:45.198 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:45.199 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:45.199 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:45.199 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:45.199 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:45.200 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:45.200 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:45.200 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:45.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:45.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:45.201 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:05:45.201 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:45.201 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:45.201 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:45.202 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:45.202 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:45.202 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:45.202 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:45.203 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:45.203 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:45.203 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:45.203 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:05:45.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:45.204 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:05:45.204 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:05:45.204 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:05:45.204 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:45.205 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:05:45.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:05:50.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:50.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:50.208 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:05:50.259 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:05:50.259 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:05:50.326 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:05:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:05:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:05:50.327 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:05:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:05:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:05:50.328 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:05:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:05:50.329 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:05:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:50.330 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:05:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:05:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:50.331 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:05:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:50.332 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:05:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:50.333 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:05:50.334 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:50.334 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:50.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:50.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:50.335 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:05:50.335 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:50.335 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:50.335 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:50.335 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:50.336 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:50.336 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:50.336 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:50.336 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:50.336 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:50.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:50.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:50.337 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:05:50.337 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:50.338 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:50.338 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:50.338 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:50.338 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:50.338 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:50.339 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:50.339 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:50.339 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:50.339 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:50.339 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:05:50.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:50.340 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:05:50.340 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:05:50.340 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:05:50.340 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:50.341 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:05:50.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:05:55.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:55.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:05:55.344 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:05:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:05:55.400 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:05:55.451 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:05:55.452 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:05:55.452 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:05:55.452 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:05:55.453 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:05:55.453 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:05:55.453 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:55.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:55.454 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:05:55.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:55.454 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:05:55.455 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:55.455 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:55.455 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:05:55.455 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:55.456 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:05:55.456 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:55.456 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:05:55.456 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:55.457 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:55.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:55.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:55.457 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:05:55.458 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:55.458 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:55.458 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:05:55.458 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:55.459 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:55.459 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:55.459 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:05:55.459 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:55.460 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:55.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:55.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:55.460 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:05:55.461 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:55.461 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:55.461 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:55.461 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:55.462 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:55.462 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:55.462 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:55.462 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:55.463 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:55.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:55.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:05:55.464 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:05:55.464 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:05:55.464 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:05:55.464 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:05:55.465 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:05:55.465 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:05:55.465 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:05:55.466 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:05:55.466 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:05:55.466 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:55.466 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:55.467 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:05:55.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:05:55.467 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:05:55.467 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:05:55.468 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:05:55.468 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:05:55.468 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:05:55.470 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:06:00.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:00.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:00.472 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:00.522 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:00.523 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:00.581 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:00.582 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:00.582 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:00.582 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:00.583 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:00.583 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:00.583 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:00.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:00.584 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:00.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:00.584 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:00.584 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:00.584 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:00.585 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:00.585 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:00.585 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:00.585 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:00.585 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:00.586 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:00.586 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:00.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:00.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:00.587 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:00.587 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:00.587 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:00.587 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:06:00.588 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:00.588 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:00.588 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:00.588 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:06:00.589 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:00.589 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:00.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:00.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:00.590 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:00.590 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:00.590 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:00.590 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:00.590 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:00.591 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:00.591 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:00.591 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:00.591 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:00.592 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:00.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:00.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:00.592 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:00.592 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:00.593 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:00.593 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:00.593 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:00.593 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:00.594 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:00.594 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:00.594 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:00.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:00.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:00.595 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:00.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:00.595 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:00.595 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:00.596 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:00.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:00.596 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:00.597 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.089761_243.1 2021-09-17T02:06:05.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:05.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:05.600 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:05.649 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:05.650 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:05.706 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:05.707 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:05.707 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:05.707 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:05.707 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:05.708 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:05.708 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:05.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:05.708 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:05.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:05.709 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:05.709 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:05.709 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:05.709 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:05.710 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:05.710 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:05.710 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:05.710 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:05.710 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:05.711 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:05.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:05.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:05.711 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:05.711 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:05.712 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:05.712 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:06:05.712 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:05.712 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:05.713 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:05.713 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:06:05.713 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:05.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:05.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:05.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:05.714 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:05.714 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:05.714 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:05.714 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:05.715 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:05.715 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:05.715 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:05.715 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:05.715 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:05.716 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:05.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:05.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:05.716 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:05.717 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:05.717 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:05.717 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:05.717 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:05.717 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:05.718 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:05.718 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:05.718 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:05.718 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:05.718 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:05.719 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:05.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:05.719 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:05.719 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:05.720 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:05.720 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:05.720 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:05.721 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.089761_243.1 2021-09-17T02:06:10.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:10.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:10.724 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:10.795 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:10.796 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:11.270 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:11.270 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:11.270 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:11.270 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:11.271 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:11.271 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:11.271 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:11.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:11.272 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:11.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:11.272 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:11.272 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:11.272 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:11.273 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:11.273 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:11.273 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:11.273 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:11.274 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:11.274 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:11.274 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:11.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:11.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:11.275 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:11.275 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:11.275 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:11.275 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:06:11.276 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:11.276 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:11.276 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:11.276 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:06:11.277 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:11.277 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:11.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:11.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:11.278 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:11.278 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:11.279 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:11.279 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:11.279 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:11.279 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:11.280 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:11.280 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:11.280 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:11.280 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:11.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:11.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:11.281 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:11.281 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:11.281 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:11.281 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:11.282 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:11.282 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:11.282 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:11.282 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:11.282 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:11.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:11.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:11.283 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:11.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:11.284 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:11.284 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:11.284 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:11.284 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:11.284 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:11.286 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:06:16.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:16.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:16.289 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:16.404 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:16.405 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:16.583 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:16.584 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:16.584 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:16.584 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:16.584 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:16.585 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:16.585 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:16.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:16.585 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:16.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:16.586 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:16.586 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:16.586 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:16.586 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:16.587 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:16.587 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:16.587 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:16.587 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:16.588 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:16.588 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:16.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:16.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:16.589 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:16.589 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:16.589 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:16.589 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:06:16.590 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:16.590 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:16.590 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:16.590 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:06:16.590 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:16.591 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:16.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:16.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:16.591 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:16.592 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:16.592 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:16.592 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:16.592 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:16.593 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:16.593 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:16.593 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:16.593 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:16.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:16.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:16.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:16.594 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:16.594 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:16.595 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:16.595 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:16.595 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:16.595 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:16.596 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:16.596 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:16.596 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:16.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:16.597 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:16.597 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:16.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:16.597 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:16.597 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:16.598 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:16.598 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:16.598 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:16.599 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.067914_240.1 2021-09-17T02:06:20.970 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:06:20.969+0000 7f62b31ff700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-17T02:06:21.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:21.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:21.602 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:21.648 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:21.698 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:21.699 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:21.700 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:21.701 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:21.702 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:21.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:21.704 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:06:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:06:21.705 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:21.706 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:21.707 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:21.708 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:21.709 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:21.710 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:21.711 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:21.711 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:21.711 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:21.711 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:21.712 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:21.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:21.713 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:21.714 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.088776_242.1 2021-09-17T02:06:26.715 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:26.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:26.717 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:26.765 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:26.765 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:26.820 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:26.821 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:26.821 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:26.821 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:26.821 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:26.822 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:26.822 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:26.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:26.822 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:26.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:26.823 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:26.823 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:26.823 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:26.823 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:26.824 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:26.824 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:26.824 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:26.824 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:26.825 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:26.825 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:26.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:26.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:26.825 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:26.826 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:26.826 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:26.826 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.062079_237.1", 2021-09-17T02:06:26.826 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:26.827 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:26.827 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:26.827 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.062079Z", 2021-09-17T02:06:26.827 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:26.827 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:26.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:26.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:26.828 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:26.828 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:26.828 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:26.829 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:26.829 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:26.829 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:26.829 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:26.829 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:26.830 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:26.830 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:26.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:26.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:26.831 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:26.831 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:26.831 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:26.832 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:26.832 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:26.832 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:26.832 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:26.832 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:26.833 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:26.833 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:26.833 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:26.833 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:26.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:26.834 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:26.834 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:26.834 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:26.834 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:26.835 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:26.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.062079_237.1 2021-09-17T02:06:31.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:31.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:31.839 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:31.910 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:31.910 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:32.185 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:32.186 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:32.186 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:32.186 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:32.186 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:32.187 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:32.187 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:32.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:32.187 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:32.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:32.188 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:32.188 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:32.188 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:32.188 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:32.189 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:32.189 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:32.189 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:32.189 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:32.189 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:32.190 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:32.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:32.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:32.190 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:32.191 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:32.191 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:32.191 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.091993_244.1", 2021-09-17T02:06:32.191 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:32.192 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:32.192 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:32.192 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.091993Z", 2021-09-17T02:06:32.192 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:32.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:32.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:32.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:32.194 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:32.194 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:32.194 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:32.195 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:32.195 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:32.195 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:32.195 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:32.195 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:32.196 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:32.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:32.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:32.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:32.197 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:32.197 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:32.197 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:32.197 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:32.198 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:32.198 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:32.198 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:32.198 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:32.199 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:32.199 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:32.199 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:32.199 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:32.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:32.200 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:32.200 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:32.200 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:32.200 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:32.200 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:32.202 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.091993_244.1 2021-09-17T02:06:37.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:37.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:37.204 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:37.298 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:37.342 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:37.393 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:37.393 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:37.394 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:37.394 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:37.394 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:37.394 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:37.395 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:37.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:37.395 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:37.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:37.395 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:37.396 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:37.396 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:37.396 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:37.396 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:37.397 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:37.397 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:37.397 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:37.397 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:37.397 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:37.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:37.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:37.398 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:37.398 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:37.399 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:37.399 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:06:37.399 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:37.399 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:37.399 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:37.400 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:06:37.400 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:37.400 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:37.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:37.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:37.401 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:37.401 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:37.401 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:37.402 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:37.402 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:37.402 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:37.402 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:37.402 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:37.403 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:37.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:37.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:37.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:37.404 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:37.404 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:37.404 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:37.404 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:37.404 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:37.405 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:37.405 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:37.405 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:37.405 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:37.406 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:37.406 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:37.406 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:37.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:37.406 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:37.407 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:37.407 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:37.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:37.407 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:37.408 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.088776_242.1 2021-09-17T02:06:42.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:42.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:42.411 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:42.456 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:42.506 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:42.506 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:42.507 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:42.507 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:42.507 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:42.507 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:42.508 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:42.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:42.508 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:42.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:42.508 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:42.509 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:42.509 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:42.509 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:42.509 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:42.509 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:42.510 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:42.510 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:42.510 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:42.510 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:42.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:42.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:42.511 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:42.511 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:42.511 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:42.512 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:06:42.512 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:42.512 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:42.512 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:42.513 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:06:42.513 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:42.513 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:42.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:42.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:42.514 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:42.514 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:42.514 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:42.514 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:42.515 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:42.515 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:42.515 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:42.515 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:42.515 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:42.516 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:42.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:42.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:42.516 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:42.517 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:42.517 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:42.517 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:42.517 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:42.517 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:42.518 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:42.518 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:42.518 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:42.518 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:42.519 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:42.519 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:42.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:42.519 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:42.519 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:42.520 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:42.520 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:42.520 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:42.521 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.067914_240.1 2021-09-17T02:06:47.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:47.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:47.524 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:47.573 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:47.574 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:47.630 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:47.631 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:47.631 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:47.631 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:47.631 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:47.632 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:47.632 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:47.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:47.632 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:47.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:47.633 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:47.633 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:47.633 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:47.634 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:47.634 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:47.634 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:47.634 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:47.635 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:47.635 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:47.635 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:47.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:47.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:47.636 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:47.636 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:47.636 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:47.636 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.062079_237.1", 2021-09-17T02:06:47.637 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:47.637 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:47.637 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:47.637 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.062079Z", 2021-09-17T02:06:47.638 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:47.638 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:47.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:47.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:47.638 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:47.639 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:47.639 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:47.639 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:47.640 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:47.640 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:47.641 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:47.641 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:47.641 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:47.641 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:47.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:47.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:47.642 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:47.643 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:47.643 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:47.643 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:47.643 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:47.644 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:47.644 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:47.644 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:47.644 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:47.644 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:47.645 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:47.645 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:47.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:47.645 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:47.646 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:47.646 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:47.646 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:47.646 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:47.647 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.062079_237.1 2021-09-17T02:06:52.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:52.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:52.691 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:52.741 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:52.741 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:52.799 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:52.800 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:52.800 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:52.800 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:52.800 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:52.801 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:52.801 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:52.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:52.801 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:52.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:52.802 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:52.802 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:52.802 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:52.803 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:52.803 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:52.803 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:52.803 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:52.803 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:52.804 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:52.804 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:52.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:52.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:52.805 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:52.805 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:52.805 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:52.805 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:06:52.805 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:52.806 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:52.806 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:52.806 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:06:52.806 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:52.807 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:52.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:52.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:52.807 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:52.808 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:52.808 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:52.808 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:52.808 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:52.808 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:52.809 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:52.809 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:52.809 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:52.809 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:52.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:52.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:52.810 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:52.811 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:52.811 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:52.811 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:52.811 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:52.811 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:52.812 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:52.812 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:52.812 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:52.812 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:52.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:52.813 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:52.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:52.813 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:52.814 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:52.814 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:52.814 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:52.814 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:52.816 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:06:57.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:57.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:06:57.818 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:06:57.898 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:06:57.898 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:06:57.990 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:06:57.990 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:06:57.990 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:06:57.991 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:06:57.991 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:06:57.991 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:06:57.991 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:57.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:57.992 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:06:57.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:57.992 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:06:57.993 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:57.993 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:57.993 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:06:57.993 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:57.993 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:06:57.994 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:57.994 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:06:57.994 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:57.994 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:57.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:57.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:57.995 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:06:57.995 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:57.995 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:57.996 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:06:57.996 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:57.996 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:57.996 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:57.997 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:06:57.997 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:57.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:57.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:57.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:57.998 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:06:57.998 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:57.998 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:57.998 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:57.999 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:57.999 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:57.999 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:57.999 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:57.999 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:58.000 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:58.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:58.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:06:58.000 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:06:58.001 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:06:58.001 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:06:58.001 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:06:58.001 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:06:58.001 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:06:58.002 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:06:58.002 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:06:58.002 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:06:58.002 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:58.003 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:58.003 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:06:58.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:06:58.003 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:06:58.003 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:06:58.004 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:06:58.004 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:06:58.004 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:06:58.005 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.088776_242.1 2021-09-17T02:07:03.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:03.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:03.009 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:07:03.052 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:07:03.053 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:07:03.103 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:07:03.104 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:07:03.104 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:07:03.104 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:07:03.105 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:07:03.105 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:07:03.106 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:03.106 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:03.107 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:07:03.107 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:03.107 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:07:03.107 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:03.108 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:03.108 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:07:03.108 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:03.109 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:07:03.109 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:03.109 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:07:03.110 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:03.110 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:03.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:03.110 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:03.111 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:07:03.111 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:03.111 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:03.111 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:07:03.112 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:03.112 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:03.112 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:03.112 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:07:03.113 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:03.113 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:03.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:03.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:03.113 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:07:03.114 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:03.114 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:03.114 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:03.114 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:03.115 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:03.115 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:03.115 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:03.115 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:03.116 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:03.116 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:03.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:03.116 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:07:03.117 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:03.117 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:03.117 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:03.118 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:03.118 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:03.118 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:03.118 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:03.119 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:03.119 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:03.119 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:03.119 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:07:03.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:03.120 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:07:03.120 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:07:03.120 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:07:03.120 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:03.121 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:07:03.122 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.086132_241.1 2021-09-17T02:07:08.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:08.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:08.125 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:07:08.176 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:07:08.176 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:07:08.230 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:07:08.231 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:07:08.231 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:07:08.231 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:07:08.231 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:07:08.232 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:07:08.232 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:08.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:08.233 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:07:08.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:08.234 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:07:08.234 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:08.234 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:08.234 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:07:08.235 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:08.235 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:07:08.235 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:08.236 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:07:08.236 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:08.236 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:08.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:08.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:08.237 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:07:08.238 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:08.238 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:08.238 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:07:08.238 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:08.239 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:08.239 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:08.239 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:07:08.239 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:08.239 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:08.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:08.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:08.240 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:07:08.240 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:08.240 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:08.241 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:08.241 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:08.241 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:08.241 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:08.242 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:08.242 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:08.242 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:08.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:08.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:08.243 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:07:08.243 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:08.243 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:08.243 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:08.244 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:08.244 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:08.245 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:08.245 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:08.245 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:08.245 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:08.246 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:08.246 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:07:08.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:08.246 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:07:08.246 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:07:08.247 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:07:08.247 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:08.247 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:07:08.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.089761_243.1 2021-09-17T02:07:13.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:13.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:13.252 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:07:13.304 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:07:13.305 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:07:13.364 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:07:13.364 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:07:13.365 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:07:13.365 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:07:13.365 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:07:13.365 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:07:13.366 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:13.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:13.366 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:07:13.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:13.367 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:07:13.367 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:13.367 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:13.367 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:07:13.367 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:13.368 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:07:13.368 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:13.368 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:07:13.368 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:13.369 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:13.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:13.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:13.369 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:07:13.369 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:13.370 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:13.370 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:07:13.370 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:13.370 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:07:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:13.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:13.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:13.372 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:07:13.372 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:13.372 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:13.373 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:13.374 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:07:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:13.375 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:13.376 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:13.377 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:13.377 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:07:13.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:13.377 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:07:13.377 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:07:13.378 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:07:13.378 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:13.378 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:07:13.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:07:18.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:18.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:18.382 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:07:18.486 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:07:18.487 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:07:18.836 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:07:18.836 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:07:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:07:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:07:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:07:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:07:18.837 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:18.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:18.838 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:07:18.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:18.838 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:07:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:07:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:18.839 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:07:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:07:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:18.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:18.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:18.841 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:07:18.841 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:18.841 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:07:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:18.842 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:07:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:18.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:07:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:18.844 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:18.845 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:18.846 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:07:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:18.847 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:18.848 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:18.849 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:07:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:07:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:07:18.850 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:07:18.851 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:18.851 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:07:18.852 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:07:23.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:23.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:23.856 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:07:23.921 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:07:23.921 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:07:23.971 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:07:23.972 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:07:23.972 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:07:23.973 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:07:23.973 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:07:23.974 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:07:23.974 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:23.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:23.975 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:07:23.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:23.976 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:07:23.976 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:23.976 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:23.977 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:07:23.977 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:23.977 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:07:23.978 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:23.978 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:07:23.979 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:23.979 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:23.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:23.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:23.980 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:07:23.981 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:23.981 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:23.981 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:07:23.982 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:23.982 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:23.982 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:23.983 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:07:23.983 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:23.984 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:23.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:23.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:23.985 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:07:23.985 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:23.986 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:23.986 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:23.986 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:23.987 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:23.987 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:23.987 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:23.987 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:23.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:23.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:23.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:23.988 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:07:23.989 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:23.989 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:23.989 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:23.989 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:23.989 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:23.990 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:23.990 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:23.990 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:23.990 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:23.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:23.991 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:07:23.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:23.991 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:07:23.991 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:07:23.992 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:07:23.992 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:23.992 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:07:23.993 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:07:28.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:28.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:28.996 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:07:29.046 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:07:29.047 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:07:29.104 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:07:29.104 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:07:29.105 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:07:29.105 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:07:29.105 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:07:29.105 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:07:29.105 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:29.106 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:29.106 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:07:29.106 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:29.106 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:07:29.107 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:29.107 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:29.107 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:07:29.107 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:29.108 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:07:29.108 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:29.108 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:07:29.108 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:29.108 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:29.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:29.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:29.109 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:07:29.109 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:29.109 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:29.110 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:07:29.110 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:29.110 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:29.110 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:29.111 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:07:29.111 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:29.111 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:29.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:29.112 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:29.112 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:07:29.112 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:29.112 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:29.113 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:29.113 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:29.113 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:29.113 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:29.113 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:29.114 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:29.114 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:29.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:29.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:29.115 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:07:29.115 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:29.115 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:29.115 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:29.115 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:29.116 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:29.116 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:29.116 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:29.116 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:29.116 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:29.117 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:29.117 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:07:29.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:29.117 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:07:29.117 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:07:29.118 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:07:29.118 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:29.118 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:07:29.120 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:07:34.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:34.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:34.123 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:07:34.174 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:07:34.175 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:07:34.232 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:07:34.232 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:07:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:07:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:07:34.233 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:07:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:07:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:07:34.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:07:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:34.235 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:07:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:07:34.236 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:34.237 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:07:34.237 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:34.237 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:34.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:34.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:34.238 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:07:34.238 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:07:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:34.239 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:07:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:34.240 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:07:34.241 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:34.242 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:34.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:07:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:34.244 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:34.245 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:07:34.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:07:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:07:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:07:34.247 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:34.247 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:07:34.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.048044_234.1 2021-09-17T02:07:39.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:39.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:39.253 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:07:39.357 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:07:39.358 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:07:39.500 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:07:39.500 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:07:39.500 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:07:39.500 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:07:39.501 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:07:39.501 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:07:39.501 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:39.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:39.502 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:07:39.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:39.503 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:07:39.503 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:39.503 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:39.504 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:07:39.504 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:39.504 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:07:39.504 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:39.504 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:07:39.505 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:39.505 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:39.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:39.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:39.506 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:07:39.506 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:39.507 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:39.507 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:07:39.507 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:39.507 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:39.507 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:39.508 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:07:39.508 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:39.508 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:39.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:39.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:39.509 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:07:39.509 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:39.509 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:39.509 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:39.510 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:39.510 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:39.510 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:39.510 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:39.511 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:39.511 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:39.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:39.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:39.511 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:07:39.512 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:39.512 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:39.512 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:39.513 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:39.513 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:39.513 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:39.513 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:39.513 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:39.514 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:39.514 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:39.514 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:07:39.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:39.515 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:07:39.515 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:07:39.515 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:07:39.515 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:39.516 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:07:39.517 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.089761_243.1 2021-09-17T02:07:44.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:44.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:44.520 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:07:44.567 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:07:44.568 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:07:44.618 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:07:44.619 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:07:44.619 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:07:44.619 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:07:44.620 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:07:44.620 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:07:44.620 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:44.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:44.620 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:07:44.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:44.621 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:07:44.621 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:44.621 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:44.622 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:07:44.622 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:44.622 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:07:44.622 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:44.623 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:07:44.623 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:44.623 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:44.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:44.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:44.624 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:07:44.624 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:44.624 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:44.624 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:07:44.625 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:44.625 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:44.625 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:44.625 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:07:44.625 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:44.626 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:44.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:44.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:44.626 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:07:44.627 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:44.627 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:44.627 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:44.627 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:44.627 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:44.628 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:44.628 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:44.628 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:44.628 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:44.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:44.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:44.629 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:07:44.629 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:44.629 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:44.630 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:44.630 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:44.630 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:44.630 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:44.631 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:44.631 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:44.631 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:44.631 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:44.631 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:07:44.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:44.632 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:07:44.632 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:07:44.632 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:07:44.632 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:44.633 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:07:44.634 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:07:49.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:49.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:49.637 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:07:49.688 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:07:49.689 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:07:49.748 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:07:49.749 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:07:49.750 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:07:49.750 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:07:49.750 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:07:49.751 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:07:49.751 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:49.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:49.752 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:07:49.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:49.754 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:07:49.754 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:49.754 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:49.755 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:07:49.755 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:49.756 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:07:49.756 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:49.757 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:07:49.757 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:49.757 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:49.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:49.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:49.759 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:07:49.759 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:49.759 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:49.760 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:07:49.760 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:49.760 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:49.761 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:49.761 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:07:49.762 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:49.762 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:49.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:49.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:49.765 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:07:49.765 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:49.765 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:49.766 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:49.766 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:49.767 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:49.768 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:49.768 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:49.769 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:49.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:49.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:49.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:49.769 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:07:49.770 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:49.770 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:49.770 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:49.777 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:49.777 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:49.777 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:49.778 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:49.778 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:49.778 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:49.778 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:49.779 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:07:49.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:49.779 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:07:49.779 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:07:49.779 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:07:49.780 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:49.780 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:07:49.781 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:07:54.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:54.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:54.785 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:07:54.837 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:07:54.838 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:07:54.895 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:07:54.896 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:07:54.896 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:07:54.897 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:07:54.897 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:07:54.897 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:07:54.897 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:54.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:54.898 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:07:54.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:54.898 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:07:54.898 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:07:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:07:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:07:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:07:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:54.902 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:54.902 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:07:54.903 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:54.903 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:54.903 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:54.903 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:07:54.903 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:54.904 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:54.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:54.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:54.904 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:07:54.905 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:54.905 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:54.905 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:54.905 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:54.906 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:54.906 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:54.906 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:54.906 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:54.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:54.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:54.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:07:54.907 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:07:54.907 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:07:54.908 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:07:54.908 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:07:54.908 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:07:54.908 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:07:54.909 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:07:54.909 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:07:54.909 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:07:54.909 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:54.909 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:54.910 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:07:54.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:07:54.910 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:07:54.910 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:07:54.911 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:07:54.911 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:07:54.911 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:07:54.912 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.065659_239.1 2021-09-17T02:07:59.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:59.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:07:59.915 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:00.006 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:00.007 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:00.299 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:00.299 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:00.300 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:00.300 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:00.300 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:00.300 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:00.301 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:00.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:00.301 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:00.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:00.302 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:00.302 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:00.303 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:00.303 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:00.303 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:00.303 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:00.304 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:00.304 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:00.304 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:00.305 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:00.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:00.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:00.306 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:00.306 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:00.306 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:00.307 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:08:00.307 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:00.307 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:00.307 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:00.308 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:08:00.308 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:00.308 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:00.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:00.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:00.309 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:00.310 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:00.310 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:00.311 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:00.311 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:00.312 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:00.312 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:00.312 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:00.313 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:00.313 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:00.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:00.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:00.315 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:00.315 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:00.316 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:00.316 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:00.317 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:00.317 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:00.317 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:00.318 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:00.318 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:00.319 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:00.319 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:00.320 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:00.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:00.320 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:00.321 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:00.321 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:00.321 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:00.322 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:00.322 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.065659_239.1 2021-09-17T02:08:05.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:05.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:05.325 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:05.399 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:05.399 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:05.526 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:05.527 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:05.527 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:05.528 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:05.528 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:05.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:05.529 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:05.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:05.530 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:05.530 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:05.530 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:05.530 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:05.530 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:05.531 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:05.531 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:05.531 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:05.531 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:05.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:05.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:05.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:05.532 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:05.532 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:05.533 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:05.533 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:08:05.533 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:05.533 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:05.533 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:05.534 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:08:05.534 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:05.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:05.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:05.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:05.535 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:05.535 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:05.535 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:05.535 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:05.536 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:05.536 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:05.536 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:05.536 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:05.536 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:05.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:05.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:05.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:05.537 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:05.537 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:05.538 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:05.538 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:05.538 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:05.539 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:05.539 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:05.540 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:05.540 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:05.540 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:05.540 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:05.541 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:05.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:05.542 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:05.542 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:05.542 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:05.543 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:05.543 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:05.543 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.269640_253.1 2021-09-17T02:08:10.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:10.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:10.546 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:10.594 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:10.595 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:10.649 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:10.649 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:10.649 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:10.650 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:10.650 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:10.650 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:10.650 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:10.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:10.651 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:10.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:10.651 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:10.651 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:10.652 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:10.652 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:10.652 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:10.652 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:10.653 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:10.653 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:10.653 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:10.653 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:10.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:10.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:10.654 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:10.654 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:10.654 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:10.655 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.157882_247.1", 2021-09-17T02:08:10.655 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:10.655 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:10.655 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:10.655 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.157882Z", 2021-09-17T02:08:10.656 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:10.656 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:10.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:10.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:10.656 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:10.657 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:10.657 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:10.657 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:10.657 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:10.658 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:10.658 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:10.658 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:10.659 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:10.659 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:10.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:10.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:10.660 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:10.660 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:10.660 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:10.660 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:10.660 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:10.661 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:10.661 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:10.661 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:10.661 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:10.661 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:10.662 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:10.662 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:10.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:10.662 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:10.663 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:10.663 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:10.663 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:10.663 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:10.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.157882_247.1 2021-09-17T02:08:15.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:15.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:15.668 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:15.719 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:15.780 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:15.780 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:15.780 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:15.780 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:15.781 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:15.781 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:15.781 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:15.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:15.782 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:15.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:15.782 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:15.782 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:15.783 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:15.783 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:15.783 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:15.784 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:15.784 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:15.784 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:15.784 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:15.785 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:15.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:15.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:15.786 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:15.786 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:15.786 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:15.787 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:08:15.787 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:15.787 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:15.787 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:15.788 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:08:15.788 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:15.788 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:15.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:15.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:15.789 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:15.789 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:15.789 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:15.790 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:15.790 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:15.790 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:15.790 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:15.791 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:15.791 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:15.791 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:15.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:15.794 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:15.794 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:15.794 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:15.794 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:15.795 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:15.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:15.795 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:15.795 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:15.796 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:15.796 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:15.797 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:15.798 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.065659_239.1 2021-09-17T02:08:20.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:20.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:20.801 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:20.892 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:20.892 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:21.191 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:21.191 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:21.191 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:21.192 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:21.192 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:21.192 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:21.192 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:21.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:21.193 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:21.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:21.193 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:21.194 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:21.194 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:21.194 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:21.194 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:21.195 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:21.195 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:21.195 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:21.195 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:21.195 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:21.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:21.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:21.196 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:21.196 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:21.197 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:21.197 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:08:21.197 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:21.197 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:21.197 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:21.198 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:08:21.198 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:21.198 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:21.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:21.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:21.199 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:21.199 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:21.199 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:21.199 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:21.200 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:21.200 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:21.200 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:21.200 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:21.200 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:21.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:21.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:21.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:21.201 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:21.201 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:21.202 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:21.202 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:21.202 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:21.202 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:21.203 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:21.203 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:21.203 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:21.203 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:21.203 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:21.204 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:21.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:21.204 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:21.204 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:21.204 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:21.205 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:21.205 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:21.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:08:26.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:26.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:26.209 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:26.290 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:26.291 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:26.364 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:26.365 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:26.365 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:26.365 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:26.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:26.366 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:26.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:26.367 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:26.367 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:26.367 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:26.367 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:26.368 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:26.368 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:26.368 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:26.368 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:26.368 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:26.369 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:26.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:26.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:26.369 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:26.369 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:26.370 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:26.370 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:08:26.370 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:26.370 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:26.371 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:26.371 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:08:26.371 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:26.371 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:26.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:26.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:26.372 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:26.372 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:26.372 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:26.372 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:26.373 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:26.373 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:26.373 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:26.373 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:26.374 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:26.374 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:26.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:26.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:26.374 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:26.375 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:26.375 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:26.375 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:26.375 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:26.375 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:26.376 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:26.376 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:26.376 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:26.376 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:26.376 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:26.377 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:26.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:26.377 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:26.377 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:26.377 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:26.378 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:26.378 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:26.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.067914_240.1 2021-09-17T02:08:31.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:31.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:31.382 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:31.432 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:31.432 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:31.491 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:31.492 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:31.493 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:31.493 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:31.494 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:31.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:31.494 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:31.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:31.495 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:31.495 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:31.496 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:31.496 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:31.496 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:31.496 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:31.496 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:31.497 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:31.497 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:31.497 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:31.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:31.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:31.498 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:31.498 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:31.499 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:31.499 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:08:31.499 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:31.499 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:31.500 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:31.500 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:08:31.500 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:31.500 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:31.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:31.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:31.501 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:31.501 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:31.501 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:31.502 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:31.502 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:31.502 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:31.503 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:31.503 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:31.503 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:31.504 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:31.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:31.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:31.504 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:31.505 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:31.505 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:31.505 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:31.506 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:31.506 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:31.506 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:31.506 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:31.507 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:31.507 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:31.507 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:31.507 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:31.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:31.508 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:31.508 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:31.508 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:31.508 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:31.509 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:31.510 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:08:36.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:36.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:36.513 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:36.567 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:36.625 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:36.625 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:36.625 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:36.625 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:36.626 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:36.626 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:36.626 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:36.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:36.627 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:36.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:36.627 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:36.627 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:36.628 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:36.628 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:36.628 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:36.628 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:36.628 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:36.629 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:36.629 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:36.629 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:36.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:36.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:36.630 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:36.630 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:36.630 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:36.630 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.157882_247.1", 2021-09-17T02:08:36.631 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:36.631 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:36.631 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:36.632 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.157882Z", 2021-09-17T02:08:36.632 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:36.632 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:36.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:36.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:36.633 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:36.634 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:36.634 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:36.635 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:36.635 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:36.635 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:36.635 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:36.636 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:36.636 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:36.636 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:36.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:36.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:36.637 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:36.637 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:36.637 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:36.638 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:36.638 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:36.638 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:36.638 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:36.638 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:36.639 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:36.639 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:36.639 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:36.639 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:36.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:36.640 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:36.640 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:36.640 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:36.641 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:36.641 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:36.642 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.157882_247.1 2021-09-17T02:08:41.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:41.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:41.645 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:41.732 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:42.077 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:42.077 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:42.078 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:42.078 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:42.078 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:42.078 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:42.079 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:42.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:42.079 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:42.079 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:42.080 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:42.080 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:42.080 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:42.080 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:42.081 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:42.081 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:42.081 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:42.081 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:42.082 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:42.082 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:42.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:42.082 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:42.082 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:42.083 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:42.083 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:42.083 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:08:42.083 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:42.084 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:42.084 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:42.084 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:08:42.084 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:42.085 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:42.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:42.086 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:42.086 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:42.086 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:42.086 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:42.087 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:42.087 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:42.087 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:42.087 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:42.088 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:42.088 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:42.088 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:42.088 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:42.088 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:42.089 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:42.089 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:42.089 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:42.089 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:42.090 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:42.090 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:42.090 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:42.091 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:42.091 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:42.091 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:42.091 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:42.092 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:42.092 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:42.092 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:42.092 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:42.093 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:42.093 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:42.093 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:42.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.086132_241.1 2021-09-17T02:08:47.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:47.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:47.100 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:47.170 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:47.171 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:47.278 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:47.279 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:47.279 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:47.280 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:47.280 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:47.281 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:47.282 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:47.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:47.283 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:47.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:47.283 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:47.284 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:47.284 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:47.285 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:47.285 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:47.285 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:47.286 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:47.286 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:47.287 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:47.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:47.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:47.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:47.289 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:47.289 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:47.289 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:47.290 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:08:47.290 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:47.291 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:47.291 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:47.292 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:08:47.292 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:47.293 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:47.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:47.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:47.294 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:47.294 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:47.295 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:47.295 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:47.295 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:47.296 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:47.296 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:47.297 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:47.298 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:47.298 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:47.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:47.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:47.299 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:47.299 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:47.299 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:47.300 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:47.300 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:47.300 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:47.300 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:47.301 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:47.301 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:47.301 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:47.301 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:47.301 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:47.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:47.302 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:47.302 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:47.303 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:47.303 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:47.303 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:47.305 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.088776_242.1 2021-09-17T02:08:52.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:52.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:52.308 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:52.359 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:52.360 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:52.418 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.062079_237.1", 2021-09-17T02:08:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.062079Z", 2021-09-17T02:08:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:52.432 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:52.432 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:52.432 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:52.435 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:52.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:52.435 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:52.436 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.062079_237.1 2021-09-17T02:08:57.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:57.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:08:57.439 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:08:57.492 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:08:57.492 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:08:57.550 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:08:57.550 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:08:57.550 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:08:57.551 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:08:57.551 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:08:57.551 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:08:57.551 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:57.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:57.552 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:08:57.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:57.552 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:08:57.553 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:57.553 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:57.553 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:08:57.553 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:57.554 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:08:57.554 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:57.554 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:08:57.554 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:57.555 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:57.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:57.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:57.555 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:08:57.556 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:57.556 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:57.556 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:08:57.556 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:57.557 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:57.557 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:57.557 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:08:57.557 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:57.558 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:57.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:57.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:57.559 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:08:57.559 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:57.560 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:57.560 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:57.560 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:57.561 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:57.561 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:57.561 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:57.561 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:57.561 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:57.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:57.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:08:57.562 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:08:57.562 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:08:57.563 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:08:57.563 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:08:57.563 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:08:57.564 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:08:57.564 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:08:57.564 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:08:57.564 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:08:57.565 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:57.565 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:57.565 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:08:57.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:08:57.565 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:08:57.566 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:08:57.566 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:08:57.566 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:08:57.566 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:08:57.568 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:09:02.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:02.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:02.571 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:02.670 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:02.671 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:02.926 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:09:02.927 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:09:02.927 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:09:02.928 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:09:02.928 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:09:02.928 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:09:02.929 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:02.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:02.930 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:09:02.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:02.930 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:09:02.931 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:02.931 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:02.931 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:02.932 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:02.932 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:09:02.933 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:02.933 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:09:02.933 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:02.934 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:02.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:02.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:02.934 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:09:02.935 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:02.935 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:02.935 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:09:02.935 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:02.935 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:02.936 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:02.936 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:09:02.936 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:02.936 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:02.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:02.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:02.937 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:09:02.937 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:02.938 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:02.938 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:02.938 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:02.938 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:02.938 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:02.939 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:02.939 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:02.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:02.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:02.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:02.940 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:09:02.940 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:02.940 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:02.940 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:02.941 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:02.941 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:02.941 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:02.942 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:02.942 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:02.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:02.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:02.943 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:09:02.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:02.943 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:09:02.943 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:09:02.944 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:09:02.944 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:02.944 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:09:02.946 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.048044_234.1 2021-09-17T02:09:07.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:07.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:07.950 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:08.027 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:08.027 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:08.158 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:09:08.158 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:09:08.158 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:09:08.159 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:09:08.159 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:09:08.159 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:09:08.159 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:08.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:08.160 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:09:08.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:08.160 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:09:08.161 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:08.161 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:08.161 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:08.161 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:08.162 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:09:08.163 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:08.163 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:09:08.163 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:08.163 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:08.163 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:08.164 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:08.164 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:09:08.164 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:08.165 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:08.165 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.212386_250.1", 2021-09-17T02:09:08.165 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:08.165 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:08.166 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:08.166 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.212386Z", 2021-09-17T02:09:08.166 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:08.166 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:08.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:08.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:08.167 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:09:08.167 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:08.168 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:08.168 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:08.168 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:08.168 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:09:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:09:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:09:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:09:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:09:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:08.176 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:09:08.176 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.212386_250.1 2021-09-17T02:09:13.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:13.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:13.179 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:13.228 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:13.228 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:13.284 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:09:13.284 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:09:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:09:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:09:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:09:13.285 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:09:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:09:13.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:09:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:13.287 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:09:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:09:13.288 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:09:13.289 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:09:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:13.290 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:09:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:13.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:09:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:13.292 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:13.293 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:13.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:09:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:13.295 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:13.296 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:09:13.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:09:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:09:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:09:13.298 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:13.298 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:09:13.300 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:09:18.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:18.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:18.303 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:18.414 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:09:18.415 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:09:18.415 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:09:18.415 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:09:18.416 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:09:18.416 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:09:18.416 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:18.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:18.417 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:09:18.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:18.417 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:09:18.417 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:18.417 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:18.418 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:18.419 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:18.419 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:09:18.419 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:18.420 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:09:18.420 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:18.420 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:18.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:18.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:18.421 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:09:18.422 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:18.422 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:18.422 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:09:18.422 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:18.423 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:18.423 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:18.423 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:09:18.423 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:18.424 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:18.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:18.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:18.424 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:09:18.425 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:18.425 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:18.425 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:18.425 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:18.425 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:18.426 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:18.426 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:18.426 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:18.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:18.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:18.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:18.427 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:09:18.427 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:18.427 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:18.428 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:18.428 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:18.428 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:18.429 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:18.429 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:18.430 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:18.430 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:18.430 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:18.430 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:09:18.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:18.431 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:09:18.432 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:09:18.432 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:09:18.432 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:18.432 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:09:18.434 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.067914_240.1 2021-09-17T02:09:23.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:23.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:23.436 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:23.518 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:23.519 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:23.792 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:09:23.792 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:09:23.792 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:09:23.793 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:09:23.793 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:09:23.794 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:09:23.794 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:23.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:23.795 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:09:23.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:23.796 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:09:23.796 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:23.797 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:23.799 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:23.800 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:23.800 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:09:23.800 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:23.801 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:09:23.802 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:23.802 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:23.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:23.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:23.803 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:09:23.803 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:23.803 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:23.804 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:09:23.804 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:23.804 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:23.804 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:23.805 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:09:23.805 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:23.805 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:23.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:23.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:23.806 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:09:23.806 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:23.806 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:23.807 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:23.807 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:23.807 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:23.807 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:23.807 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:23.808 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:23.808 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:23.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:23.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:23.808 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:09:23.809 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:23.809 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:23.809 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:23.810 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:23.810 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:23.810 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:23.811 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:23.811 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:23.811 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:23.811 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:23.811 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:09:23.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:23.812 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:09:23.812 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:09:23.812 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:09:23.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:23.813 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:09:23.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.067914_240.1 2021-09-17T02:09:28.818 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:28.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:28.820 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:28.864 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:28.864 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:28.915 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:09:28.915 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:09:28.915 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:09:28.916 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:09:28.916 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:09:28.916 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:09:28.916 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:28.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:28.917 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:09:28.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:28.917 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:09:28.917 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:28.918 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:28.918 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:28.918 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:28.918 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:09:28.919 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:28.919 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:09:28.919 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:28.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:28.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:28.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:28.920 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:09:28.920 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:28.920 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:28.921 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:09:28.922 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:28.922 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:28.922 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:28.922 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:09:28.923 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:28.923 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:28.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:28.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:28.924 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:09:28.924 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:28.924 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:28.924 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:28.924 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:28.925 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:28.925 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:28.925 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:28.925 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:28.927 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:28.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:28.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:28.931 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:09:28.931 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:28.932 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:28.932 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:28.932 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:28.932 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:28.933 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:28.933 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:28.933 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:28.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:28.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:28.934 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:09:28.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:28.934 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:09:28.934 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:09:28.934 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:09:28.935 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:28.935 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:09:28.937 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844261.911090_301.1 target=1_1631844259.049023_235.1 2021-09-17T02:09:33.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2021-09-17T02:09:33.940 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-39 2021-09-17T02:09:33.964 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-40 2021-09-17T02:09:34.001 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-41 2021-09-17T02:09:34.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:34.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:34.035 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:34.083 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:34.083 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:34.134 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:09:34.148 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:09:34.148 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:09:34.149 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:34.149 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:09:34.149 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:09:34.149 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:09:34.150 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844574.020970_329.1", 2021-09-17T02:09:34.150 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:09:34.020970Z" 2021-09-17T02:09:34.150 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:09:34.150 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:09:34.151 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:09:34.151 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:09:34.151 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:09:34.151 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:09:34.152 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:09:34.152 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:09:34.152 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:09:34.152 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:09:34.154 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:09:34.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:34.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:34.156 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:34.205 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:34.206 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:34.268 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:09:34.268 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:09:34.268 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:09:34.269 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:09:34.269 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:09:34.269 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:09:34.269 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:34.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:34.270 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:09:34.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:34.270 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:09:34.271 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:34.271 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:34.271 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:34.271 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:34.272 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:09:34.272 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:34.272 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:09:34.272 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:34.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:34.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:34.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:34.273 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:09:34.274 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:34.274 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:34.274 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.062079_237.1", 2021-09-17T02:09:34.274 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:34.275 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:34.275 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:34.276 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.062079Z", 2021-09-17T02:09:34.276 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:34.277 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:34.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:34.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:34.278 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:09:34.278 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:34.278 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:34.279 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:34.279 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:34.279 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:34.279 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:34.280 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:34.280 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:34.280 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:34.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:34.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:34.281 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:09:34.281 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:34.281 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:34.282 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:34.282 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:34.282 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:34.282 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:34.282 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:34.283 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:34.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:34.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:34.283 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:09:34.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:34.284 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:09:34.284 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:09:34.284 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:09:34.284 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:34.285 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:09:34.286 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.062079_237.1 2021-09-17T02:09:39.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:39.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:39.290 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:39.340 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:39.341 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:39.398 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:09:39.398 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:09:39.399 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:09:39.399 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:09:39.399 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:09:39.399 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:09:39.400 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:39.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:39.400 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:09:39.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:39.401 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:09:39.401 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:39.401 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:39.401 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:39.401 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:39.402 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:09:39.402 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:39.402 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:09:39.402 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:39.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:39.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:39.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:39.403 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:09:39.403 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:39.404 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:39.404 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:09:39.404 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:39.404 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:39.405 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:39.405 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:09:39.405 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:39.406 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:39.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:39.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:39.406 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:09:39.407 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:39.407 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:39.407 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:39.407 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:39.408 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:39.408 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:39.408 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:39.408 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:39.408 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:39.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:39.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:39.409 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:09:39.409 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:39.409 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:39.410 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:39.411 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:09:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:39.412 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:09:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:09:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:09:39.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:39.413 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:09:39.415 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.048044_234.1 2021-09-17T02:09:44.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:44.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:44.418 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:44.502 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:44.502 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:44.811 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:09:44.812 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:09:44.812 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:09:44.812 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:09:44.813 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:09:44.813 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:09:44.813 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:44.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:44.814 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:09:44.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:44.814 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:09:44.814 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:44.815 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:44.815 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:44.815 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:44.815 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:09:44.815 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:44.816 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:09:44.816 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:44.816 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:44.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:44.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:44.817 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:09:44.817 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:44.817 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:44.818 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.209532_249.1", 2021-09-17T02:09:44.818 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:44.818 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:44.818 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:44.819 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.209532Z", 2021-09-17T02:09:44.819 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:44.819 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:44.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:44.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:44.820 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:09:44.820 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:44.820 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:44.820 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:44.821 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:44.821 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:44.821 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:44.821 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:44.821 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:44.822 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:44.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:44.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:44.822 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:09:44.822 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:44.823 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:44.824 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:09:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:09:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:09:44.825 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:09:44.826 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:44.826 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:09:44.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.209532_249.1 2021-09-17T02:09:49.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:49.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:49.831 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:49.884 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:49.885 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:49.937 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:09:49.937 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:09:49.937 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:09:49.938 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:09:49.938 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:09:49.938 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:09:49.938 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:49.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:49.939 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:09:49.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:49.939 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:09:49.939 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:49.940 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:49.940 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:49.940 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:49.940 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:09:49.941 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:49.941 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:09:49.941 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:49.941 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:49.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:49.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:49.942 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:09:49.942 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:49.942 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:49.942 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:09:49.943 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:49.943 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:49.943 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:49.943 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:09:49.944 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:49.944 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:49.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:49.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:49.944 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:09:49.945 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:49.945 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:49.945 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:49.945 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:49.945 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:49.946 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:49.946 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:49.946 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:49.946 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:49.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:49.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:49.947 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:09:49.947 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:49.948 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:49.948 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:49.948 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:49.948 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:49.948 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:49.949 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:49.949 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:49.949 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:49.949 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:49.949 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:09:49.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:49.950 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:09:49.950 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:09:49.950 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:09:49.951 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:49.951 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:09:49.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.049023_235.1 2021-09-17T02:09:54.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:54.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:09:54.956 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:09:55.004 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:09:55.004 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:09:55.056 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:09:55.056 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:09:55.057 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:09:55.057 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:09:55.057 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:09:55.057 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:09:55.057 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:55.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:55.058 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:09:55.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:55.058 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:09:55.059 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:55.059 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:55.059 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:09:55.059 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:55.060 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:09:55.060 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:55.060 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:09:55.060 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:55.060 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:55.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:55.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:55.061 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:09:55.061 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:55.062 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:55.062 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:09:55.062 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:55.062 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:55.062 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:55.063 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:09:55.063 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:55.064 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:55.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:55.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:55.064 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:09:55.064 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:55.065 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:55.065 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:55.065 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:55.065 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:55.066 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:55.066 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:55.066 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:55.066 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:55.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:55.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:09:55.067 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:09:55.067 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:09:55.067 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:09:55.068 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:09:55.068 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:09:55.068 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:09:55.068 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:09:55.068 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:09:55.069 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:09:55.069 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:55.069 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:55.069 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:09:55.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:09:55.070 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:09:55.070 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:09:55.070 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:09:55.070 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:09:55.071 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:09:55.072 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.049023_235.1 2021-09-17T02:10:00.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:00.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:00.074 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:00.120 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:00.120 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:00.173 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:00.173 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:00.173 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:00.173 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:00.174 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:00.174 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:00.174 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:00.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:00.175 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:00.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:00.175 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:00.175 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:00.175 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:00.176 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:00.176 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:00.176 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:00.176 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:00.177 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:00.177 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:00.177 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:00.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:00.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:00.178 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:00.178 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:00.178 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:00.178 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:10:00.179 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:00.179 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:00.179 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:00.179 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:10:00.180 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:00.180 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:00.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:00.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:00.181 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:00.181 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:00.181 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:00.181 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:00.182 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:00.182 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:00.182 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:00.182 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:00.182 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:00.183 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:00.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:00.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:00.183 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:00.183 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:00.184 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:00.184 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:00.184 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:00.184 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:00.184 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:00.185 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:00.185 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:00.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:00.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:00.186 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:00.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:00.186 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:00.186 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:00.186 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:00.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:00.187 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:00.188 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.048044_234.1 2021-09-17T02:10:05.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:05.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:05.192 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:05.242 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:05.243 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:05.301 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:05.301 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:05.302 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:05.302 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:05.302 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:05.302 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:05.303 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:05.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:05.303 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:05.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:05.303 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:05.304 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:05.305 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:05.305 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:05.305 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:05.305 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:05.306 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:05.306 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:05.306 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:05.306 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:05.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:05.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:05.307 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:05.307 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:05.307 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:05.308 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:10:05.308 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:05.308 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:05.308 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:05.309 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:10:05.310 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:05.311 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:05.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:05.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:05.312 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:05.312 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:05.312 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:05.312 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:05.313 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:05.313 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:05.313 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:05.313 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:05.313 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:05.314 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:05.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:05.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:05.317 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:05.317 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:05.317 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:05.318 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:05.318 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:05.318 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:05.318 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:05.319 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:05.320 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:05.320 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:05.320 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:05.321 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:05.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:05.321 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:05.321 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:05.322 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:05.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:05.322 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:05.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.065659_239.1 2021-09-17T02:10:10.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:10.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:10.326 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:10.377 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:10.377 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:10.438 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:10.438 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:10.439 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:10.439 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:10.439 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:10.439 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:10.439 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:10.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:10.440 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:10.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:10.440 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:10.441 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:10.441 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:10.441 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:10.443 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:10.443 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:10.443 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:10.444 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:10.444 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:10.444 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:10.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:10.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:10.445 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:10.445 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:10.446 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:10.446 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:10:10.447 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:10.447 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:10.447 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:10.447 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:10:10.448 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:10.448 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:10.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:10.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:10.449 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:10.449 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:10.449 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:10.449 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:10.449 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:10.450 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:10.450 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:10.450 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:10.450 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:10.451 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:10.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:10.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:10.451 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:10.452 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:10.452 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:10.452 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:10.452 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:10.453 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:10.453 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:10.453 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:10.453 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:10.454 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:10.454 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:10.454 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:10.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:10.455 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:10.455 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:10.455 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:10.455 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:10.456 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:10.457 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.089761_243.1 2021-09-17T02:10:15.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:15.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:15.461 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:15.532 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:15.533 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:16.009 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:16.010 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:16.010 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:16.010 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:16.011 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:16.011 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:16.011 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:16.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:16.011 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:16.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:16.012 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:16.012 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:16.012 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:16.013 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:16.013 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:16.013 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:16.013 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:16.014 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:16.014 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:16.014 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:16.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:16.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:16.015 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:16.015 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:16.015 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:16.015 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:10:16.016 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:16.016 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:16.016 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:16.016 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:10:16.017 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:16.017 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:16.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:16.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:16.017 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:16.018 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:16.018 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:16.018 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:16.018 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:16.019 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:16.019 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:16.019 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:16.019 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:16.019 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:16.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:16.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:16.020 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:16.020 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:16.021 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:16.021 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:16.021 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:16.021 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:16.021 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:16.022 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:16.022 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:16.022 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:16.022 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:16.023 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:16.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:16.023 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:16.023 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:16.024 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:16.024 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:16.024 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:16.025 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.065659_239.1 2021-09-17T02:10:21.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:21.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:21.028 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:21.088 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:21.089 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:21.143 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:21.143 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:21.144 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:21.144 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:21.144 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:21.144 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:21.144 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:21.145 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:21.146 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:21.148 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:21.148 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:21.148 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:21.149 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:21.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:21.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:21.149 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:21.150 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:21.150 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:21.150 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:10:21.150 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:21.151 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:21.151 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:21.151 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:10:21.152 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:21.152 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:21.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:21.152 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:21.152 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:21.153 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:21.153 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:21.153 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:21.153 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:21.153 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:21.154 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:21.154 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:21.154 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:21.154 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:21.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:21.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:21.155 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:21.155 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:21.155 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:21.156 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:21.156 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:21.156 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:21.156 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:21.157 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:21.157 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:21.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:21.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:21.158 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:21.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:21.158 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:21.158 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:21.159 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:21.159 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:21.159 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:21.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.067914_240.1 2021-09-17T02:10:26.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:26.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:26.164 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:26.207 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:26.258 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:26.259 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:26.259 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:26.259 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:26.259 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:26.259 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:26.260 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:26.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:26.260 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:26.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:26.261 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:26.261 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:26.261 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:26.261 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:26.261 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:26.262 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:26.262 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:26.262 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:26.262 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:26.263 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:26.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:26.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:26.264 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:26.264 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:26.264 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:26.264 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:10:26.265 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:26.265 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:26.265 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:26.265 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:10:26.265 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:26.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:26.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:26.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:26.267 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:26.267 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:26.268 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:26.268 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:26.268 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:26.268 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:26.269 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:26.269 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:26.269 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:26.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:26.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:26.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:26.271 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:26.271 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:26.271 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:26.271 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:26.272 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:26.272 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:26.272 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:26.272 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:26.272 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:26.273 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:26.273 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:26.273 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:26.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:26.274 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:26.274 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:26.274 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:26.274 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:26.275 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:26.276 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.269640_253.1 2021-09-17T02:10:31.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:31.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:31.279 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:31.324 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:31.324 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:31.377 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:31.378 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:31.378 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:31.378 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:31.378 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:31.379 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:31.379 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:31.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:31.379 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:31.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:31.380 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:31.380 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:31.380 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:31.381 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:31.381 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:31.381 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:31.382 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:31.383 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:31.383 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:31.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:31.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:31.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:31.384 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:31.384 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:31.384 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:31.385 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.103309_245.1", 2021-09-17T02:10:31.385 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:31.385 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:31.385 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:31.386 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.103309Z", 2021-09-17T02:10:31.386 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:31.386 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:31.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:31.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:31.387 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:31.387 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:31.387 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:31.388 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:31.388 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:31.388 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:31.388 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:31.389 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:31.389 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:31.389 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:31.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:31.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:31.390 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:31.390 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:31.390 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:31.391 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:31.391 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:31.391 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:31.391 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:31.392 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:31.392 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:31.392 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:31.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:31.393 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:31.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:31.393 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:31.394 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:31.394 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:31.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:31.394 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:31.396 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.103309_245.1 2021-09-17T02:10:36.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:36.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:36.399 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:36.449 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:36.450 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:36.509 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:36.509 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:36.510 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:36.510 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:36.510 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:36.510 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:36.511 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:36.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:36.511 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:36.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:36.512 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:36.512 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:36.512 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:36.512 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:36.513 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:36.513 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:36.513 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:36.513 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:36.514 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:36.514 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:36.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:36.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:36.515 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:36.515 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:36.515 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:36.515 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:10:36.516 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:36.516 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:36.516 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:36.516 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:10:36.517 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:36.517 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:36.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:36.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:36.518 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:36.518 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:36.518 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:36.518 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:36.518 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:36.519 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:36.519 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:36.519 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:36.519 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:36.520 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:36.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:36.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:36.520 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:36.521 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:36.521 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:36.521 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:36.521 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:36.521 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:36.522 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:36.523 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:36.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:36.525 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:36.526 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.049023_235.1 2021-09-17T02:10:41.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:41.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:41.528 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:41.580 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:41.581 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:41.643 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:41.643 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:41.644 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:41.644 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:41.644 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:41.644 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:41.645 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:41.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:41.645 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:41.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:41.646 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:41.646 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:41.646 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:41.646 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:41.647 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:41.647 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:41.647 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:41.647 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:41.647 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:41.648 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:41.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:41.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:41.648 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:41.649 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:41.649 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:41.649 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:10:41.649 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:41.649 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:41.650 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:41.650 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:10:41.650 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:41.651 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:41.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:41.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:41.651 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:41.652 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:41.652 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:41.652 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:41.652 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:41.653 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:41.653 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:41.653 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:41.653 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:41.653 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:41.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:41.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:41.654 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:41.655 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:41.655 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:41.655 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:41.656 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:41.656 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:41.656 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:41.656 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:41.656 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:41.657 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:41.657 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:41.657 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:41.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:41.657 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:41.658 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:41.658 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:41.658 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:41.658 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:41.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.089761_243.1 2021-09-17T02:10:46.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:46.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:46.662 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:46.710 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:46.711 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:46.768 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:46.769 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:46.769 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:46.769 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:46.769 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:46.770 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:46.770 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:46.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:46.770 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:46.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:46.771 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:46.771 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:46.771 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:46.771 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:46.772 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:46.772 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:46.772 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:46.772 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:46.772 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:46.773 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:46.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:46.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:46.773 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:46.774 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:46.774 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:46.774 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.103309_245.1", 2021-09-17T02:10:46.774 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:46.775 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:46.775 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:46.775 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.103309Z", 2021-09-17T02:10:46.775 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:46.775 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:46.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:46.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:46.776 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:46.777 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:46.777 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:46.778 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:46.778 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:46.778 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:46.778 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:46.779 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:46.779 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:46.779 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:46.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:46.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:46.780 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:46.780 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:46.780 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:46.780 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:46.781 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:46.781 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:46.781 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:46.782 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:46.782 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:46.782 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:46.782 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:46.783 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:46.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:46.783 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:46.783 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:46.784 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:46.784 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:46.784 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:46.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.103309_245.1 2021-09-17T02:10:51.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:51.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:51.788 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:51.864 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:51.864 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:52.337 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.288419_257.1", 2021-09-17T02:10:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:52.344 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:52.344 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:52.344 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.288419Z", 2021-09-17T02:10:52.344 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:52.344 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:52.351 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:52.351 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:52.351 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:52.351 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:52.353 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.288419_257.1 2021-09-17T02:10:57.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:57.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:10:57.355 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:10:57.419 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:10:57.419 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:10:57.558 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:10:57.559 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:10:57.559 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:10:57.559 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:10:57.560 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:10:57.561 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:10:57.561 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:57.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:57.561 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:10:57.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:57.563 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:10:57.563 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:57.563 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:57.564 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:10:57.564 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:57.564 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:10:57.564 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:57.565 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:10:57.565 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:57.565 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:57.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:57.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:57.566 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:10:57.566 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:57.566 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:57.566 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.157882_247.1", 2021-09-17T02:10:57.567 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:57.567 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:57.567 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:57.567 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.157882Z", 2021-09-17T02:10:57.567 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:57.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:57.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:57.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:57.568 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:10:57.569 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:57.569 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:57.569 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:57.569 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:57.569 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:57.570 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:57.570 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:57.570 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:57.570 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:57.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:57.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:10:57.571 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:10:57.571 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:10:57.572 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:10:57.573 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:10:57.573 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:10:57.573 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:10:57.573 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:10:57.574 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:10:57.574 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:10:57.574 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:57.574 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:57.575 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:10:57.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:10:57.575 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:10:57.575 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:10:57.575 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:10:57.576 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:10:57.576 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:10:57.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.157882_247.1 2021-09-17T02:11:02.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:02.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:02.580 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:02.623 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:02.624 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:02.676 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:02.676 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:02.677 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:02.677 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:02.677 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:02.677 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:02.678 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:02.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:02.678 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:02.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:02.679 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:02.679 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:02.679 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:02.679 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:02.679 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:02.680 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:02.680 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:02.681 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:02.681 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:02.681 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:02.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:02.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:02.682 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:02.682 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:11:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:02.683 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:02.684 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:11:02.684 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:02.684 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:02.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:02.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:02.685 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:02.685 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:02.685 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:02.685 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:02.686 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:02.686 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:02.686 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:02.686 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:02.687 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:02.687 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:02.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:02.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:02.687 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:02.688 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:02.688 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:02.688 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:02.688 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:02.689 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:02.689 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:02.689 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:02.689 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:02.689 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:02.690 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:02.690 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:02.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:02.690 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:02.691 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:02.691 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:02.691 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:02.691 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:02.692 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.065659_239.1 2021-09-17T02:11:07.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:07.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:07.695 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:07.740 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:07.743 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:07.793 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:07.794 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:07.794 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:07.795 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:07.795 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:07.796 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:07.796 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:07.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:07.797 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:07.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:07.797 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:07.797 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:07.797 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:07.798 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:07.798 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:07.798 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:07.798 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:07.799 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:07.799 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:07.799 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:07.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:07.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:07.800 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:07.800 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:07.800 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:07.800 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:11:07.800 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:07.801 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:07.801 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:07.801 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:11:07.801 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:07.802 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:07.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:07.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:07.802 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:07.802 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:07.803 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:07.803 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:07.803 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:07.803 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:07.804 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:07.804 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:07.805 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:07.805 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:07.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:07.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:07.806 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:07.806 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:07.806 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:07.806 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:07.806 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:07.807 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:07.807 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:07.807 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:07.807 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:07.808 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:07.808 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:07.808 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:07.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:07.808 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:07.809 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:07.809 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:07.809 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:07.809 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:07.811 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.086132_241.1 2021-09-17T02:11:12.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:12.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:12.814 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:12.864 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:12.864 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:12.922 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.062079_237.1", 2021-09-17T02:11:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.062079Z", 2021-09-17T02:11:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:12.939 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:12.941 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.062079_237.1 2021-09-17T02:11:17.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:17.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:17.944 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:18.056 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:18.056 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:18.057 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:18.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:18.059 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:18.060 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:18.061 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.051622_236.1", 2021-09-17T02:11:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:18.062 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.051622Z", 2021-09-17T02:11:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:18.063 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:18.064 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:18.065 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:18.066 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:18.067 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:18.068 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:18.069 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:18.070 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:18.070 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:18.072 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.051622_236.1 2021-09-17T02:11:23.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:23.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:23.075 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:23.126 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:23.127 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:23.184 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:23.184 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:23.185 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:23.185 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:23.185 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:23.185 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:23.186 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:23.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:23.186 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:23.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:23.187 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:23.187 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:23.187 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:23.187 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:23.187 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:23.188 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:23.188 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:23.188 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:23.188 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:23.189 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:23.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:23.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:23.189 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:23.189 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:23.190 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:23.190 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.161433_248.1", 2021-09-17T02:11:23.191 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:23.191 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:23.191 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:23.191 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.161433Z", 2021-09-17T02:11:23.192 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:23.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:23.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:23.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:23.192 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:23.193 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:23.193 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:23.193 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:23.193 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:23.194 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:23.194 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:23.194 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:23.194 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:23.195 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:23.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:23.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:23.196 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:23.196 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:23.196 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:23.197 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:23.197 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:23.197 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:23.197 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:23.197 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:23.198 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:23.198 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:23.198 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:23.198 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:23.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:23.199 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:23.199 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:23.199 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:23.199 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:23.200 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:23.202 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.161433_248.1 2021-09-17T02:11:28.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:28.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:28.205 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:28.258 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:28.258 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:28.406 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:28.406 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:28.406 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:28.407 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:28.407 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:28.407 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:28.407 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:28.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:28.408 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:28.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:28.408 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:28.409 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:28.409 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:28.409 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:28.409 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:28.410 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:28.410 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:28.411 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:28.411 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:28.412 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:28.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:28.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:28.413 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:28.414 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:28.414 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:28.415 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.161433_248.1", 2021-09-17T02:11:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:28.417 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.161433Z", 2021-09-17T02:11:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:28.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:28.419 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:28.420 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:28.420 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:28.420 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:28.421 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:28.421 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:28.421 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:28.421 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:28.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:28.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:28.422 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:28.423 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:28.423 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:28.424 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:28.424 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:28.425 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:28.425 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:28.425 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:28.426 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:28.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:28.427 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:28.427 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:28.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:28.428 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:28.428 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:28.428 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:28.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:28.428 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:28.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.161433_248.1 2021-09-17T02:11:33.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:33.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:33.433 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:33.519 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:33.519 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:33.840 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:33.840 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:33.840 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:33.841 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:33.841 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:33.841 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:33.841 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:33.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:33.842 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:33.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:33.842 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:33.842 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:33.843 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:33.843 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:33.843 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:33.843 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:33.844 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:33.844 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:33.844 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:33.844 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:33.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:33.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:33.845 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:33.845 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:33.845 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:33.846 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.051622_236.1", 2021-09-17T02:11:33.846 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:33.846 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:33.846 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:33.846 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.051622Z", 2021-09-17T02:11:33.847 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:33.847 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:33.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:33.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:33.847 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:33.848 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:33.848 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:33.848 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:33.848 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:33.849 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:33.849 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:33.849 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:33.849 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:33.849 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:33.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:33.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:33.850 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:33.850 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:33.851 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:33.851 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:33.851 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:33.851 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:33.851 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:33.852 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:33.852 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:33.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:33.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:33.852 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:33.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:33.853 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:33.853 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:33.853 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:33.853 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:33.854 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:33.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.051622_236.1 2021-09-17T02:11:38.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:38.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:38.857 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:38.932 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:38.933 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:39.073 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:39.074 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:39.075 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:39.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:39.077 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:39.078 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:39.079 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:39.079 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:39.079 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:39.079 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:39.080 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:39.080 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:39.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:39.080 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:39.081 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:39.081 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:39.081 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:39.081 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:11:39.081 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:39.082 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:39.082 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:39.082 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:11:39.082 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:39.083 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:39.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:39.084 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:39.084 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:39.084 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:39.084 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:39.085 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:39.085 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:39.086 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:39.086 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:39.086 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:39.086 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:39.086 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:39.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:39.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:39.087 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:39.087 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:39.088 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:39.088 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:39.088 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:39.088 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:39.088 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:39.089 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:39.089 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:39.089 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:39.089 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:39.089 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:39.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:39.090 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:39.090 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:39.090 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:39.090 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:39.091 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:39.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.086132_241.1 2021-09-17T02:11:44.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:44.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:44.095 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:44.139 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:44.140 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:44.191 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:44.191 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:44.192 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:44.192 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:44.192 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:44.192 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:44.193 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:44.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:44.193 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:44.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:44.194 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:44.194 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:44.194 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:44.194 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:44.195 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:44.195 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:44.195 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:44.195 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:44.196 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:44.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:44.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:44.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:44.196 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:44.197 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:44.197 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:44.197 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.062079_237.1", 2021-09-17T02:11:44.197 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:44.197 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:44.198 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:44.198 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.062079Z", 2021-09-17T02:11:44.198 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:44.198 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:44.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:44.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:44.199 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:44.199 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:44.200 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:44.200 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:44.200 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:44.200 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:44.200 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:44.201 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:44.201 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:44.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:44.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:44.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:44.202 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:44.202 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:44.202 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:44.203 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:44.203 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:44.203 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:44.203 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:44.204 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:44.204 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:44.204 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:44.204 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:44.204 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:44.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:44.205 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:44.205 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:44.205 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:44.205 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:44.206 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:44.207 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.062079_237.1 2021-09-17T02:11:49.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:49.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:49.210 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:49.261 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:49.320 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:49.320 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:49.320 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:49.321 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:49.321 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:49.321 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:49.321 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:49.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:49.322 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:49.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:49.323 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:49.323 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:49.323 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:49.324 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:49.324 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:49.324 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:49.324 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:49.325 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:49.325 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:49.325 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:49.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:49.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:49.326 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:49.326 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:49.326 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:49.326 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:11:49.327 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:49.327 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:49.327 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:49.327 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:11:49.328 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:49.328 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:49.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:49.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:49.329 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:49.330 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:49.330 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:49.330 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:49.330 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:49.331 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:49.331 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:49.331 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:49.331 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:49.332 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:49.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:49.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:49.333 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:49.333 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:49.333 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:49.334 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:49.334 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:49.334 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:49.334 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:49.335 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:49.335 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:49.335 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:49.335 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:49.336 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:49.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:49.336 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:49.336 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:49.336 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:49.337 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:49.337 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:49.338 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.064665_238.1 2021-09-17T02:11:54.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:54.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:54.342 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:54.395 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:54.395 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:54.453 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:54.453 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:54.453 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:54.454 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:54.454 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:54.454 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:54.454 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:54.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:54.455 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:54.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:54.455 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:54.455 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:54.456 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:54.456 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:54.456 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:54.456 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:54.456 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:54.457 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:54.457 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:54.457 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:54.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:54.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:54.458 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:54.458 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:54.458 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:54.458 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:11:54.459 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:54.459 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:54.459 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:54.459 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:11:54.459 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:54.460 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:54.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:54.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:54.460 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:54.461 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:54.461 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:54.461 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:54.461 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:54.461 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:54.462 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:54.462 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:54.462 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:54.462 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:54.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:54.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:54.463 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:54.463 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:54.464 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:54.464 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:54.464 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:54.464 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:54.464 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:54.465 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:54.465 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:54.465 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:54.465 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:54.465 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:54.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:54.466 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:54.466 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:54.466 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:54.467 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:54.467 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:54.468 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.065659_239.1 2021-09-17T02:11:59.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:59.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:11:59.472 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:11:59.522 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:11:59.523 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:11:59.580 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:11:59.581 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:11:59.581 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:11:59.582 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:11:59.582 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:11:59.582 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:11:59.583 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:59.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:59.583 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:11:59.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:59.583 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:11:59.584 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:59.584 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:59.584 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:11:59.584 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:59.585 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:11:59.585 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:59.585 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:11:59.585 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:59.585 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:59.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:59.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:59.586 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:11:59.586 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:59.586 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:59.587 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:11:59.587 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:59.587 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:59.587 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:59.588 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:11:59.588 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:59.588 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:59.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:59.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:59.589 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:11:59.589 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:59.589 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:59.589 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:59.589 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:59.590 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:59.590 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:59.590 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:59.590 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:59.590 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:59.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:59.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:11:59.591 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:11:59.592 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:11:59.592 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:11:59.593 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:11:59.593 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:11:59.593 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:11:59.593 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:11:59.594 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:11:59.594 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:11:59.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:59.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:59.594 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:11:59.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:11:59.595 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:11:59.595 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:11:59.595 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:11:59.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:11:59.596 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:11:59.597 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.067914_240.1 2021-09-17T02:12:04.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:04.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:04.601 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:12:04.658 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:12:04.659 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:12:05.021 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:12:05.022 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:12:05.022 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:12:05.022 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:12:05.023 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:12:05.023 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:12:05.023 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:05.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:05.023 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:12:05.034 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:05.034 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:12:05.034 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:05.034 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:05.035 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:12:05.035 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:05.035 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:12:05.035 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:05.036 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:12:05.036 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:05.036 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:05.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:05.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:05.037 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:12:05.046 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:05.046 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:05.046 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.091993_244.1", 2021-09-17T02:12:05.047 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:05.047 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:05.048 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:05.049 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.091993Z", 2021-09-17T02:12:05.049 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:05.049 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:05.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:05.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:05.050 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:12:05.051 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:05.051 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:05.051 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:05.052 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:05.052 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:05.053 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:05.054 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:05.054 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:05.055 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:05.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:05.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:05.055 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:12:05.055 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:05.056 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:05.056 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:05.056 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:05.056 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:05.057 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:05.057 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:05.057 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:05.057 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:05.057 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:05.058 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:12:05.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:05.059 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:12:05.059 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:12:05.059 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:12:05.059 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:05.059 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:12:05.060 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.091993_244.1 2021-09-17T02:12:10.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:10.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:10.064 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:12:10.116 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:12:10.116 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:12:10.172 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:12:10.172 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:12:10.173 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:12:10.173 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:12:10.173 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:12:10.173 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:12:10.173 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:10.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:10.174 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:12:10.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:10.174 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:12:10.175 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:10.175 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:10.175 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:12:10.175 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:10.175 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:12:10.176 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:10.176 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:12:10.176 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:10.176 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:10.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:10.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:10.177 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:12:10.177 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:10.177 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:10.180 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.062079_237.1", 2021-09-17T02:12:10.180 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:10.180 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:10.180 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:10.180 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.062079Z", 2021-09-17T02:12:10.181 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:10.181 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:10.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:10.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:10.182 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:12:10.182 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:10.182 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:10.183 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:10.183 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:10.183 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:10.183 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:10.184 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:10.184 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:10.184 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:10.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:10.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:10.185 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:12:10.185 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:10.185 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:10.185 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:10.185 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:10.186 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:10.186 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:10.186 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:10.186 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:10.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:10.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:10.187 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:12:10.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:10.187 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:12:10.188 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:12:10.188 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:12:10.188 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:10.189 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:12:10.190 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.062079_237.1 2021-09-17T02:12:15.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:15.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:15.412 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:12:15.458 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:12:15.459 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:12:15.512 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:12:15.512 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:12:15.512 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:12:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:12:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:12:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:12:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:15.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:12:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:15.514 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:12:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:12:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:15.515 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:12:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:12:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:15.516 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:12:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.313372_261.1", 2021-09-17T02:12:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.313372Z", 2021-09-17T02:12:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:12:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:12:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:15.526 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:15.526 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:15.526 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:15.526 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:15.527 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:15.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:15.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:15.527 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:12:15.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:15.528 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:12:15.528 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:12:15.528 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:12:15.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:15.528 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:12:15.530 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.313372_261.1 2021-09-17T02:12:20.532 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:20.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:20.534 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:12:20.586 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:12:20.586 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:12:20.644 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:12:20.644 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:12:20.645 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:12:20.645 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:12:20.645 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:12:20.645 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:12:20.646 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:20.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:20.646 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:12:20.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:20.646 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:12:20.647 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:20.647 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:20.647 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:12:20.647 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:20.649 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:12:20.649 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:20.649 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:12:20.649 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:20.650 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:20.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:20.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:20.650 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:12:20.650 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:20.651 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:20.651 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:12:20.651 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:20.651 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:20.652 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:20.652 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:12:20.652 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:20.653 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:20.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:20.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:20.654 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:12:20.654 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:20.654 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:20.654 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:20.655 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:20.655 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:20.655 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:20.655 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:20.656 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:20.656 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:20.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:20.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:20.659 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:12:20.659 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:20.659 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:20.660 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:20.660 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:20.660 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:20.660 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:20.661 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:20.661 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:20.661 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:20.661 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:20.665 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:12:20.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:20.665 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:12:20.666 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:12:20.666 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:12:20.666 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:20.667 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:12:20.667 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.049023_235.1 2021-09-17T02:12:25.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:25.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:25.671 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:12:25.723 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:12:25.723 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:12:25.783 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:12:25.784 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:12:25.784 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:12:25.784 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:12:25.784 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:12:25.785 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:12:25.785 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:25.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:25.786 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:12:25.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:25.786 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:12:25.786 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:25.786 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:25.787 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:12:25.787 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:25.788 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:12:25.788 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:25.789 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:12:25.789 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:25.789 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:25.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:25.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:25.790 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:12:25.790 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:25.790 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:25.790 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:12:25.791 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:25.791 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:25.791 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:25.791 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:12:25.792 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:25.792 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:25.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:25.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:25.794 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:12:25.794 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:25.794 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:25.794 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:25.795 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:25.795 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:25.795 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:25.795 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:25.795 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:25.796 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:25.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:25.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:25.796 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:12:25.796 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:25.797 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:25.797 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:25.797 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:25.797 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:25.798 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:25.798 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:25.799 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:25.799 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:25.799 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:25.799 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:12:25.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:25.800 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:12:25.800 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:12:25.801 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:12:25.801 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:25.802 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:12:25.802 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.049023_235.1 2021-09-17T02:12:30.805 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:30.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:30.807 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:12:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:12:30.857 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:12:30.915 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:12:30.915 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:12:30.915 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:12:30.916 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:12:30.916 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:12:30.916 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:12:30.916 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:30.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:30.917 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:12:30.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:30.917 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:12:30.917 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:30.918 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:30.918 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:12:30.918 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:30.918 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:12:30.918 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:30.919 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:12:30.919 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:30.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:30.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:30.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:30.920 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:12:30.920 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:30.920 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:30.920 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:12:30.921 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:30.921 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:30.921 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:30.922 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:12:30.922 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:30.922 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:30.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:30.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:30.923 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:12:30.923 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:30.924 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:30.924 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:30.924 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:30.924 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:30.925 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:30.925 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:30.925 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:30.925 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:30.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:30.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:30.926 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:12:30.926 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:30.926 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:30.927 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:30.927 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:30.927 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:30.927 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:30.927 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:30.928 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:30.928 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:30.928 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:30.928 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:12:30.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:30.929 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:12:30.929 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:12:30.929 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:12:30.929 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:30.930 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:12:30.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.048044_234.1 2021-09-17T02:12:35.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:35.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:35.934 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:12:36.038 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:12:36.038 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:12:36.345 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:12:36.346 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:12:36.347 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:12:36.348 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:12:36.348 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:12:36.348 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:12:36.349 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:36.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:36.350 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:12:36.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:36.350 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:12:36.350 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:36.351 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:36.351 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:12:36.351 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:36.351 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:12:36.351 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:36.352 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:12:36.352 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:36.352 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:36.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:36.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:36.353 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:12:36.353 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:36.353 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:36.354 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.062079_237.1", 2021-09-17T02:12:36.354 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:36.354 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:36.354 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:36.354 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.062079Z", 2021-09-17T02:12:36.355 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:36.355 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:36.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:36.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:36.356 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:12:36.356 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:36.356 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:36.356 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:36.356 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:36.357 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:36.357 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:36.357 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:36.357 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:36.357 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:36.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:36.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:36.358 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:12:36.358 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:36.359 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:36.359 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:36.359 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:36.360 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:36.360 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:36.361 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:36.361 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:36.361 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:36.361 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:36.362 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:12:36.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:36.362 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:12:36.362 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:12:36.362 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:12:36.363 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:36.363 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:12:36.364 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.062079_237.1 2021-09-17T02:12:41.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:41.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:41.369 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:12:41.417 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:12:41.418 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:12:41.562 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:12:41.563 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:12:41.563 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:12:41.563 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:12:41.564 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:12:41.564 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:12:41.564 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:41.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:41.565 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:12:41.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:41.565 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:12:41.565 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:41.566 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:41.566 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:12:41.566 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:41.567 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:12:41.567 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:41.567 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:12:41.568 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:41.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:41.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:41.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:41.569 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:12:41.569 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:41.569 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:41.569 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:12:41.569 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:41.570 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:41.570 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:41.570 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:12:41.570 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:41.571 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:41.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:41.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:41.571 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:12:41.572 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:41.572 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:41.572 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:41.572 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:41.572 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:41.573 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:41.573 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:41.573 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:41.573 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:41.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:41.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:41.574 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:12:41.574 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:41.574 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:41.575 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:41.575 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:41.575 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:41.575 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:41.575 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:41.576 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:41.576 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:41.576 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:41.576 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:12:41.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:41.577 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:12:41.577 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:12:41.578 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:12:41.578 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:41.578 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:12:41.578 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.086132_241.1 2021-09-17T02:12:46.098 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T02:12:46.097+0000 7f4365fc9700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-17T02:12:46.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:46.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:46.584 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:12:46.628 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:12:46.629 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:12:46.679 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:12:46.680 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:12:46.680 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:12:46.680 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:12:46.681 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:12:46.681 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:12:46.681 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:46.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:46.681 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:12:46.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:46.682 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:12:46.682 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:46.683 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:46.687 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:12:46.687 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:46.694 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:12:46.695 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:46.695 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:12:46.695 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:46.696 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:46.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:46.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:46.696 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:12:46.697 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:46.697 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:46.697 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.091993_244.1", 2021-09-17T02:12:46.711 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:46.711 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:46.711 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:46.712 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.091993Z", 2021-09-17T02:12:46.712 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:46.712 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:46.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:46.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:46.713 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:12:46.713 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:46.713 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:46.713 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:46.714 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:46.714 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:46.714 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:46.716 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:46.717 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:46.718 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:46.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:46.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:46.719 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:12:46.719 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:46.720 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:46.720 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:46.720 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:46.720 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:46.721 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:46.721 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:46.721 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:46.721 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:46.721 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:46.722 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:12:46.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:46.735 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:12:46.736 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:12:46.737 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:12:46.737 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:46.738 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:12:46.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.091993_244.1 2021-09-17T02:12:51.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:51.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:51.744 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:12:51.798 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:12:51.799 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:12:51.858 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:12:51.858 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:12:51.858 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:12:51.859 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:12:51.859 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:12:51.859 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:12:51.859 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:51.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:51.860 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:12:51.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:51.860 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:12:51.860 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:51.860 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:51.861 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:12:51.861 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:51.861 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:12:51.861 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:51.862 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:12:51.862 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:51.862 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:51.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:12:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:51.863 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:51.864 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:12:51.864 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:51.864 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:51.864 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:51.864 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:12:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:51.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:12:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:51.866 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:51.867 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:51.871 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:51.872 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:51.872 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:51.873 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:51.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:51.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:51.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:51.873 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:12:51.874 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:51.874 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:51.876 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:51.876 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:51.876 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:51.877 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:51.877 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:51.877 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:51.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:51.878 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:51.878 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:12:51.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:51.878 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:12:51.879 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:12:51.879 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:12:51.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:51.879 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:12:51.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.064665_238.1 2021-09-17T02:12:56.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:56.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:12:56.885 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:12:56.937 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:12:56.937 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:12:56.998 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:12:56.999 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:12:56.999 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:12:57.001 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:12:57.001 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:12:57.001 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:12:57.002 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:57.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:57.002 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:12:57.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:57.002 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:12:57.003 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:57.003 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:57.003 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:12:57.003 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:12:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:12:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:12:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:12:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:12:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:12:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:57.017 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:57.017 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:57.017 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:57.017 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:12:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:12:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:12:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:12:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:12:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:12:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:12:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:12:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:12:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:12:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:12:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:12:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:12:57.026 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:12:57.026 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:12:57.026 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:12:57.026 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:12:57.027 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.089761_243.1 2021-09-17T02:13:02.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:02.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:02.032 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:02.084 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:02.085 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:02.144 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:02.144 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:02.144 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:02.145 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:02.145 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:02.145 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:02.150 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:02.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:02.151 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:02.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:02.151 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:02.152 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:02.152 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:02.152 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:02.152 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:02.153 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:02.153 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:02.153 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:02.154 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:02.154 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:02.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:02.154 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:02.155 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:02.155 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:02.155 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:02.156 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.157882_247.1", 2021-09-17T02:13:02.156 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:02.156 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:02.156 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:02.156 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.157882Z", 2021-09-17T02:13:02.157 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:02.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:02.157 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:02.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:02.157 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:02.164 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:02.165 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:02.165 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:02.165 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:02.165 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:02.166 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:02.166 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:02.166 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:02.166 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:02.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:02.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:02.167 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:02.167 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:02.168 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:02.168 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:02.168 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:02.168 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:02.169 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:02.169 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:02.169 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:02.169 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:02.169 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:02.170 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:02.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:02.170 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:02.170 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:02.171 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:02.171 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:02.171 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:02.172 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.157882_247.1 2021-09-17T02:13:07.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:07.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:07.175 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:07.228 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:07.228 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:07.290 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:07.290 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:07.291 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:07.291 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:07.291 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:07.292 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:07.292 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:07.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:07.292 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:07.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:07.293 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:07.293 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:07.293 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:07.293 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:07.294 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:07.294 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:07.294 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:07.294 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:07.294 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:07.295 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:07.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:07.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:07.296 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:07.296 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:07.297 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:07.297 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.062079_237.1", 2021-09-17T02:13:07.297 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:07.297 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:07.298 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:07.298 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.062079Z", 2021-09-17T02:13:07.298 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:07.301 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:07.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:07.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:07.302 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:07.302 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:07.303 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:07.303 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:07.303 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:07.303 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:07.304 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:07.304 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:07.305 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:07.305 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:07.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:07.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:07.306 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:07.307 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:07.307 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:07.307 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:07.307 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:07.308 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:07.308 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:07.308 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:07.308 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:07.309 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:07.310 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:07.334 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:07.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:07.335 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:07.335 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:07.336 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:07.336 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:07.336 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:07.336 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.062079_237.1 2021-09-17T02:13:12.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:12.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:12.339 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:12.417 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:12.418 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:12.802 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:12.802 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:12.803 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:12.803 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:12.803 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:12.803 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:12.804 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:12.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:12.804 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:12.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:12.805 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:12.805 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:12.805 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:12.806 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:12.806 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:12.807 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:12.807 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:12.807 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:12.808 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:12.808 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:12.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:12.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:12.809 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:12.809 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:12.809 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:12.809 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:13:12.810 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:12.810 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:12.810 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:12.810 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:13:12.811 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:12.811 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:12.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:12.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:12.812 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:12.812 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:12.812 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:12.812 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:12.812 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:12.813 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:12.813 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:12.813 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:12.813 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:12.814 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:12.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:12.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:12.815 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:12.815 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:12.815 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:12.816 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:12.816 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:12.817 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:12.817 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:12.817 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:12.818 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:12.818 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:12.819 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:12.819 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:12.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:12.820 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:12.820 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:12.821 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:12.821 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:12.822 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:12.822 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.049023_235.1 2021-09-17T02:13:17.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:17.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:17.826 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:17.870 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:17.871 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:17.973 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:17.973 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:17.974 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:17.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:17.976 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:17.977 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:17.978 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:17.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:17.980 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:13:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:17.981 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:13:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:17.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:17.983 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:17.984 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:17.984 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:17.984 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:17.984 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:17.984 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:17.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:17.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:17.985 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:17.985 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:17.986 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:17.986 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:17.987 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:17.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:17.989 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:17.990 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:17.990 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:17.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.048044_234.1 2021-09-17T02:13:22.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:22.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:22.994 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:23.068 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:23.068 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:23.194 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:23.194 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:23.194 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:23.195 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:23.195 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:23.195 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:23.195 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:23.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:23.196 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:23.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:23.197 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:23.197 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:23.197 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:23.197 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:23.197 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:23.198 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:23.198 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:23.198 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:23.198 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:23.199 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:23.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:23.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:23.199 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:23.200 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:23.200 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:23.200 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:13:23.200 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:23.200 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:23.201 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:23.201 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:13:23.201 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:23.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:23.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:23.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:23.202 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:23.202 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:23.202 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:23.203 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:23.203 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:23.203 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:23.204 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:23.204 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:23.204 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:23.204 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:23.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:23.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:23.205 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:23.205 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:23.205 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:23.206 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:23.206 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:23.206 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:23.206 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:23.206 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:23.207 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:23.207 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:23.207 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:23.207 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:23.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:23.208 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:23.208 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:23.208 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:23.208 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:23.209 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:23.210 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.089761_243.1 2021-09-17T02:13:28.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:28.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:28.214 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:28.264 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:28.265 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:28.321 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:28.321 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:28.321 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:28.322 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:28.322 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:28.322 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:28.322 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:28.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:28.323 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:28.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:28.323 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:28.324 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:28.324 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:28.324 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:28.324 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:28.324 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:28.325 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:28.325 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:28.325 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:28.326 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:28.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:28.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:28.326 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:28.327 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:28.327 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:28.327 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:13:28.327 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:28.328 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:28.328 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:28.328 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:13:28.328 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:28.328 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:28.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:28.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:28.329 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:28.329 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:28.330 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:28.330 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:28.330 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:28.331 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:28.331 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:28.331 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:28.332 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:28.332 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:28.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:28.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:28.333 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:28.333 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:28.333 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:28.333 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:28.333 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:28.334 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:28.334 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:28.334 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:28.334 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:28.335 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:28.335 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:28.335 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:28.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:28.335 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:28.336 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:28.336 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:28.337 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:28.337 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:28.338 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.089761_243.1 2021-09-17T02:13:33.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:33.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:33.341 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:33.391 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:33.451 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:13:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:13:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:33.460 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:33.461 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:33.461 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:33.461 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:33.462 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:33.462 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:33.462 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:33.462 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:33.463 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:33.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:33.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:33.464 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:33.464 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:33.464 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:33.465 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:33.465 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:33.465 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:33.465 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:33.466 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:33.466 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:33.466 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:33.466 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:33.466 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:33.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:33.467 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:33.467 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:33.467 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:33.468 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:33.468 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:33.472 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.049023_235.1 2021-09-17T02:13:38.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:38.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:38.474 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:38.526 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:38.527 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:38.587 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:38.587 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:38.587 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:38.588 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:38.588 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:38.588 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:38.588 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:38.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:38.589 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:38.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:38.589 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:38.590 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:38.590 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:38.591 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:38.592 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:38.592 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:38.592 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:38.592 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:38.593 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:38.593 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:38.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:38.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:38.594 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:38.594 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:38.594 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:38.595 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:13:38.595 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:38.595 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:38.595 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:38.596 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:13:38.596 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:38.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:38.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:38.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:38.597 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:38.597 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:38.598 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:38.598 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:38.598 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:38.598 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:38.599 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:38.599 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:38.599 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:38.599 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:38.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:38.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:38.600 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:38.601 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:38.601 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:38.602 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:38.602 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:38.602 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:38.603 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:38.603 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:38.603 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:38.603 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:38.604 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:38.604 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:38.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:38.604 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:38.605 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:38.605 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:38.605 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:38.605 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:38.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.049023_235.1 2021-09-17T02:13:43.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:43.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:43.609 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:43.659 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:43.660 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:43.717 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:43.717 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:43.718 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:43.718 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:43.718 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:43.718 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:43.719 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:43.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:43.719 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:43.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:43.720 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:43.720 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:43.720 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:43.721 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:43.721 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:43.721 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:43.721 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:43.722 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:43.722 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:43.722 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:43.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:43.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:43.723 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:43.723 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:43.723 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:43.723 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:13:43.724 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:43.724 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:43.724 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:43.724 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:13:43.725 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:43.725 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:43.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:43.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:43.726 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:43.727 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:43.727 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:43.728 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:43.728 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:43.728 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:43.729 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:43.729 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:43.729 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:43.729 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:43.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:43.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:43.730 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:43.730 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:43.731 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:43.731 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:43.731 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:43.731 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:43.731 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:43.732 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:43.732 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:43.732 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:43.732 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:43.733 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:43.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:43.733 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:43.733 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:43.733 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:43.734 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:43.734 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:43.735 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.089761_243.1 2021-09-17T02:13:48.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:48.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:48.739 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:48.791 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:48.791 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:48.850 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:48.851 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:48.851 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:48.851 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:48.852 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:48.852 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:48.852 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:48.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:48.852 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:48.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:48.853 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:48.853 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:48.853 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:48.854 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:48.854 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:48.854 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:48.854 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:48.854 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:48.855 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:48.855 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:48.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:48.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:48.855 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:48.856 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:48.856 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:48.856 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:13:48.856 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:48.857 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:48.857 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:48.857 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:13:48.857 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:48.857 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:48.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:48.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:48.858 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:48.858 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:48.859 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:48.859 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:48.859 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:48.859 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:48.859 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:48.860 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:48.860 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:48.860 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:48.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:48.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:48.861 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:48.861 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:48.861 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:48.861 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:48.862 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:48.862 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:48.862 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:48.862 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:48.862 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:48.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:48.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:48.863 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:48.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:48.864 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:48.864 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:48.864 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:48.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:48.864 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:48.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.049023_235.1 2021-09-17T02:13:53.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:53.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:53.870 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:53.951 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:53.951 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:54.368 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:54.369 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:54.369 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:54.369 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:54.369 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:54.369 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:54.370 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:54.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:54.370 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:54.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:54.371 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:54.371 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:54.371 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:54.371 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:54.371 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:54.372 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:54.372 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:54.372 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:54.372 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:54.372 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:54.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:54.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:54.373 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:54.373 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:54.374 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:54.374 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:13:54.374 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:54.374 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:54.374 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:54.375 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:13:54.375 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:54.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:54.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:54.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:54.376 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:54.376 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:54.376 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:54.376 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:54.377 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:54.377 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:54.377 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:54.377 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:54.378 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:54.378 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:54.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:54.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:54.378 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:54.379 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:54.379 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:54.379 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:54.379 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:54.379 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:54.380 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:54.380 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:54.380 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:54.380 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:54.381 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:54.381 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:54.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:54.381 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:54.381 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:54.382 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:54.382 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:54.382 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:54.383 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.064665_238.1 2021-09-17T02:13:59.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:59.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:13:59.386 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:13:59.503 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:13:59.504 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:13:59.705 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:13:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:13:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:13:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:13:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:13:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:13:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:13:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:13:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:13:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:13:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:13:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:13:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:13:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:13:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:13:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:13:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:13:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:13:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:13:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:13:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:13:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:13:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:13:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:13:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:13:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:13:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:13:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:13:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:13:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:13:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:13:59.722 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:13:59.722 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.064665_238.1 2021-09-17T02:14:04.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:04.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:04.726 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:04.832 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:04.835 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:04.898 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:14:04.898 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:14:04.898 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:14:04.898 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:14:04.899 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:14:04.899 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:14:04.899 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:04.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:04.900 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:14:04.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:04.900 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:14:04.900 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:04.900 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:04.901 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:04.901 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:04.901 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:14:04.901 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:04.902 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:14:04.902 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:04.902 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:04.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:04.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:04.903 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:14:04.903 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:04.903 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:04.903 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:14:04.904 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:04.904 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:04.904 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:04.904 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:14:04.904 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:04.905 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:04.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:04.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:04.905 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:14:04.905 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:04.906 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:04.906 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:04.906 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:04.906 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:04.907 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:04.907 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:04.907 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:04.907 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:04.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:04.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:04.908 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:14:04.908 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:04.908 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:04.908 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:04.909 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:04.909 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:04.909 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:04.909 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:04.910 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:04.910 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:04.912 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:04.912 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:14:04.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:04.912 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:14:04.913 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:14:04.913 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:14:04.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:04.913 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:14:04.915 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.067914_240.1 2021-09-17T02:14:09.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:09.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:09.918 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:10.244 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:10.244 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:10.304 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:14:10.305 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:14:10.305 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:14:10.305 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:14:10.305 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:14:10.306 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:14:10.306 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:10.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:10.306 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:14:10.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:10.307 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:14:10.307 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:10.307 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:10.307 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:10.308 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:10.308 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:14:10.308 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:10.308 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:14:10.308 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:10.310 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:10.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:10.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:10.311 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:14:10.311 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:10.311 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:10.311 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:14:10.312 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:10.312 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:10.312 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:10.312 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:14:10.312 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:10.313 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:10.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:10.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:10.313 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:14:10.314 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:10.314 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:10.314 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:10.314 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:10.315 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:10.315 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:10.315 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:10.315 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:10.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:10.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:10.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:10.316 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:14:10.316 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:10.317 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:10.317 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:10.317 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:10.317 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:10.317 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:10.319 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:14:10.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:10.319 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:14:10.319 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:14:10.319 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:14:10.320 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:10.320 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:14:10.321 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.048044_234.1 2021-09-17T02:14:15.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:15.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:15.324 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:15.663 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:15.792 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:14:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:14:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:14:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:14:15.794 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:14:15.794 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:14:15.794 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:15.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:15.795 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:14:15.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:15.795 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:14:15.795 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:15.795 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:15.796 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:15.796 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:15.796 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:14:15.796 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:15.796 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:14:15.797 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:15.797 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:15.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:15.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:15.797 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:14:15.798 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:15.798 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:15.798 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:14:15.800 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:15.800 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:15.800 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:15.801 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:14:15.801 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:15.801 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:15.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:15.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:15.802 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:14:15.802 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:15.802 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:15.802 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:15.803 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:15.803 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:15.803 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:15.803 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:15.803 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:15.804 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:15.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:15.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:15.804 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:14:15.804 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:15.805 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:15.805 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:15.805 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:15.805 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:15.805 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:15.806 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:15.806 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:15.806 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:15.807 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:15.807 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:14:15.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:15.807 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:14:15.808 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:14:15.808 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:14:15.808 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:15.812 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:14:15.813 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.269640_253.1 2021-09-17T02:14:20.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:20.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:20.817 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:20.911 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:20.912 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:21.327 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:14:21.327 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:14:21.328 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:14:21.328 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:14:21.328 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:14:21.328 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:14:21.329 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:21.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:21.329 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:14:21.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:21.329 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:14:21.330 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:21.330 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:21.330 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:21.330 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:21.330 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:14:21.331 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:21.331 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:14:21.331 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:21.331 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:21.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:21.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:21.332 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:14:21.332 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:21.332 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:21.333 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.161433_248.1", 2021-09-17T02:14:21.333 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:21.333 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:21.333 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:21.333 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.161433Z", 2021-09-17T02:14:21.334 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:21.334 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:21.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:21.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:21.335 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:14:21.335 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:21.335 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:21.335 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:21.335 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:21.336 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:21.336 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:21.336 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:21.336 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:21.336 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:21.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:21.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:21.337 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:14:21.337 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:21.337 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:21.338 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:21.338 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:21.338 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:21.338 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:21.339 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:21.339 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:21.339 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:21.339 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:21.339 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:14:21.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:21.340 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:14:21.340 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:14:21.340 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:14:21.340 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:21.341 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:14:21.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.161433_248.1 2021-09-17T02:14:26.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:26.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:26.345 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:26.401 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:26.402 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:26.544 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:14:26.545 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:14:26.545 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:14:26.545 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:14:26.545 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:14:26.546 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:14:26.546 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:26.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:26.546 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:14:26.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:26.547 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:14:26.547 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:26.548 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:26.548 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:26.548 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:26.549 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:14:26.549 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:26.549 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:14:26.549 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:26.550 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:26.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:26.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:26.550 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:14:26.551 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:26.551 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:26.551 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.288419_257.1", 2021-09-17T02:14:26.551 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:26.551 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:26.552 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:26.552 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.288419Z", 2021-09-17T02:14:26.552 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:26.552 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:26.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:26.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:26.553 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:14:26.553 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:26.553 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:26.554 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:26.554 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:26.554 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:26.554 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:26.554 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:26.555 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:26.555 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:26.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:26.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:26.556 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:14:26.556 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:26.556 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:26.556 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:26.556 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:26.557 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:26.557 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:26.557 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:26.557 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:26.557 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:26.558 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:26.558 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:14:26.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:26.558 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:14:26.559 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:14:26.559 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:14:26.559 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:26.559 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:14:26.560 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.288419_257.1 2021-09-17T02:14:31.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:31.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:31.562 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:31.606 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:31.607 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:31.661 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:14:31.661 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:14:31.661 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:14:31.662 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:14:31.662 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:14:31.662 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:14:31.662 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:31.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:31.663 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:14:31.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:31.663 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:14:31.663 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:31.664 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:31.664 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:31.664 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:31.664 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:14:31.665 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:31.665 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:14:31.665 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:31.665 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:31.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:31.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:31.666 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:14:31.666 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:31.666 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:31.667 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.051622_236.1", 2021-09-17T02:14:31.667 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:31.667 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:31.667 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:31.667 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.051622Z", 2021-09-17T02:14:31.668 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:31.668 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:31.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:31.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:31.669 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:14:31.669 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:31.669 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:31.669 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:31.669 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:31.670 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:31.670 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:31.670 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:31.670 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:31.671 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:31.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:31.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:31.671 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:14:31.671 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:31.672 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:31.672 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:31.672 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:31.672 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:31.672 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:31.673 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:31.673 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:31.673 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:31.673 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:31.674 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:14:31.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:31.674 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:14:31.674 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:14:31.674 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:14:31.675 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:31.675 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:14:31.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.051622_236.1 2021-09-17T02:14:36.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:36.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:36.679 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:36.725 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:36.725 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:36.779 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:14:36.779 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:14:36.780 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:14:36.780 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:14:36.780 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:14:36.781 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:14:36.781 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:36.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:36.781 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:14:36.781 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:36.782 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:14:36.782 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:36.782 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:36.783 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:36.783 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:36.783 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:14:36.783 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:36.784 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:14:36.784 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:36.784 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:36.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:36.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:36.785 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:14:36.786 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:36.786 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:36.786 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:14:36.786 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:36.787 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:36.787 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:36.787 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:14:36.787 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:36.787 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:36.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:36.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:36.788 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:14:36.788 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:36.789 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:36.789 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:36.789 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:36.789 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:36.789 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:36.790 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:36.790 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:36.790 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:36.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:36.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:36.791 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:14:36.791 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:36.791 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:36.791 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:36.792 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:36.792 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:36.792 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:36.792 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:36.792 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:36.793 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:36.793 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:36.793 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:14:36.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:36.794 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:14:36.794 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:14:36.794 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:14:36.794 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:36.794 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:14:36.796 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.089761_243.1 2021-09-17T02:14:40.974 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:14:40.972+0000 7f62b31ff700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-17T02:14:41.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:41.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:41.798 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:41.846 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:41.846 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:41.901 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:14:41.902 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:14:41.902 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:14:41.902 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:14:41.902 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:14:41.903 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:14:41.903 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:41.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:41.903 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:14:41.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:41.904 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:14:41.904 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:41.904 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:41.904 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:41.905 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:41.905 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:14:41.905 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:41.905 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:14:41.906 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:41.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:41.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:41.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:41.907 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:14:41.907 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:41.907 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:41.907 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:14:41.907 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:41.908 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:41.908 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:41.908 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:14:41.908 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:41.908 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:41.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:41.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:41.909 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:14:41.909 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:41.910 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:41.910 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:41.910 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:41.910 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:41.910 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:41.911 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:41.911 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:41.911 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:41.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:41.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:41.912 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:14:41.912 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:41.912 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:41.912 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:41.913 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:41.913 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:41.913 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:41.913 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:41.913 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:41.914 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:41.914 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:41.914 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:14:41.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:41.915 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:14:41.915 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:14:41.915 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:14:41.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:41.915 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:14:41.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844574.020970_329.1 target=1_1631844259.049023_235.1 2021-09-17T02:14:46.919 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2021-09-17T02:14:46.920 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-42 2021-09-17T02:14:46.945 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-43 2021-09-17T02:14:47.978 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-44 2021-09-17T02:14:48.038 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:14:48.036+0000 7f62cc64b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:14:48.097 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:14:48.095+0000 7f1e2503d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:14:48.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:48.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:48.156 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:48.201 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:48.202 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:48.242 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:14:48.256 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:14:48.256 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:14:48.256 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:48.257 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:14:48.257 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:14:48.257 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:14:48.257 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844887.995835_351.1", 2021-09-17T02:14:48.258 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:14:47.995835Z" 2021-09-17T02:14:48.258 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:14:48.258 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:14:48.258 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:14:48.259 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:14:48.259 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:14:48.259 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:14:48.259 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:14:48.260 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:14:48.260 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:14:48.260 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:14:48.261 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:14:48.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:48.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:48.263 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:48.344 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:48.344 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:48.665 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:14:48.666 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:14:48.666 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:14:48.666 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:14:48.666 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:14:48.667 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:14:48.667 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:48.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:48.667 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:14:48.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:48.668 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:14:48.668 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:48.668 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:48.669 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:48.669 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:48.669 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:14:48.669 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:48.670 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:14:48.670 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:48.670 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:48.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:48.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:48.671 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:14:48.671 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:48.672 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:48.672 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:14:48.672 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:48.672 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:48.672 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:48.673 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:14:48.673 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:48.673 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:48.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:48.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:48.674 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:14:48.674 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:48.675 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:48.675 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:48.675 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:48.675 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:48.676 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:48.676 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:48.676 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:48.676 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:48.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:48.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:48.677 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:14:48.677 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:48.677 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:48.678 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:48.678 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:48.678 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:48.678 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:48.679 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:48.679 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:48.679 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:48.679 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:48.679 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:14:48.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:48.680 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:14:48.680 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:14:48.681 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:14:48.681 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:48.681 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:14:48.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.089761_243.1 2021-09-17T02:14:49.247 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:14:49.246+0000 7f62c3e3a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:14:49.437 INFO:tasks.rgw.c2.client.0.smithi045.stdout:2021-09-17T02:14:49.436+0000 7f62c8e44700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:14:49.438 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T02:14:49.437+0000 7f4374c00700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:14:53.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:53.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:53.685 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:53.771 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:53.771 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:54.023 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:14:54.023 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:14:54.024 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:14:54.024 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:14:54.024 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:14:54.025 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:14:54.025 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:54.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:54.025 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:14:54.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:54.026 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:14:54.026 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:54.026 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:54.026 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:54.027 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:54.027 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:14:54.027 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:54.027 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:14:54.028 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:54.028 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:54.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:54.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:54.029 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:14:54.029 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:54.029 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:54.029 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.051622_236.1", 2021-09-17T02:14:54.029 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:54.030 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:54.030 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:54.030 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.051622Z", 2021-09-17T02:14:54.031 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:54.032 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:54.032 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:54.032 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:54.033 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:14:54.033 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:54.033 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:54.034 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:54.034 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:54.034 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:54.035 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:54.035 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:54.035 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:54.035 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:54.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:54.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:54.036 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:14:54.036 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:54.037 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:54.037 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:54.037 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:54.037 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:54.037 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:54.038 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:54.038 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:54.038 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:54.038 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:54.038 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:14:54.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:54.039 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:14:54.039 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:14:54.039 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:14:54.040 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:54.040 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:14:54.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.051622_236.1 2021-09-17T02:14:59.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:59.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:59.044 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:59.114 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:14:59.167 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:14:59.168 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:14:59.168 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:14:59.168 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:14:59.168 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:14:59.169 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:14:59.169 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:59.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:59.169 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:14:59.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:59.170 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:14:59.170 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:59.170 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:59.171 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:14:59.171 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:59.171 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:14:59.171 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:59.171 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:14:59.172 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:59.172 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:59.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:59.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:59.173 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:14:59.173 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:59.173 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:59.173 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:14:59.173 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:59.174 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:59.174 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:59.174 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:14:59.175 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:59.175 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:59.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:59.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:59.176 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:14:59.176 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:59.177 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:59.177 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:59.178 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:59.178 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:59.178 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:59.178 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:59.183 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:59.183 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:59.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:59.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:14:59.184 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:14:59.184 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:14:59.184 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:14:59.184 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:14:59.185 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:14:59.185 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:14:59.185 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:14:59.185 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:14:59.185 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:14:59.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:59.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:59.186 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:14:59.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:14:59.186 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:14:59.187 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:14:59.187 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:14:59.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:14:59.187 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:14:59.189 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:15:04.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:04.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:04.291 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:15:04.351 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:15:04.351 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:15:04.403 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:15:04.404 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:15:04.404 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:15:04.404 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:15:04.404 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:15:04.405 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:15:04.405 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:04.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:04.405 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:15:04.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:04.406 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:15:04.406 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:04.406 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:04.406 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:15:04.407 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:04.407 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:15:04.407 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:04.407 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:15:04.407 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:04.408 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:04.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:04.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:04.408 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:15:04.408 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:04.409 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:04.409 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:15:04.409 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:04.409 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:04.409 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:04.410 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:15:04.410 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:04.410 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:04.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:04.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:04.411 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:15:04.411 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:04.411 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:04.411 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:04.412 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:04.412 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:04.412 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:04.412 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:04.412 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:04.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:04.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:04.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:04.413 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:15:04.414 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:04.414 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:04.414 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:04.414 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:04.414 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:04.415 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:04.415 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:04.415 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:04.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:04.416 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:04.416 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:15:04.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:04.416 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:15:04.416 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:15:04.417 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:15:04.417 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:04.417 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:15:04.418 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.048044_234.1 2021-09-17T02:15:09.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:09.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:09.420 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:15:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:15:09.465 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:15:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:15:09.518 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:15:09.518 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:15:09.518 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:15:09.518 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:15:09.521 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:15:09.521 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:09.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:09.522 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:15:09.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:09.522 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:15:09.522 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:09.523 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:09.523 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:15:09.523 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:09.523 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:15:09.524 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:09.524 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:15:09.524 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:09.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:09.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:09.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:09.525 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:15:09.525 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:09.525 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:09.525 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:15:09.526 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:09.526 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:09.526 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:09.526 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:15:09.527 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:09.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:09.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:09.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:09.527 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:15:09.528 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:09.528 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:09.528 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:09.528 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:09.528 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:09.529 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:09.529 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:09.529 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:09.529 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:09.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:09.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:09.530 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:15:09.530 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:09.530 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:09.531 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:09.531 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:09.531 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:09.531 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:09.531 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:09.532 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:09.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:09.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:09.532 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:15:09.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:09.533 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:15:09.533 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:15:09.533 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:15:09.533 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:09.534 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:15:09.536 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:15:14.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:14.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:14.538 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:15:14.582 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:15:14.582 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:15:14.635 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:15:14.636 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:15:14.636 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:15:14.636 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:15:14.636 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:15:14.637 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:15:14.637 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:14.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:14.637 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:15:14.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:14.638 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:15:14.638 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:14.638 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:14.638 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:15:14.639 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:14.639 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:15:14.639 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:14.639 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:15:14.640 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:14.640 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:14.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:14.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:14.640 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:15:14.641 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:14.641 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:14.641 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:15:14.641 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:14.641 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:14.642 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:14.642 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:15:14.642 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:14.642 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:14.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:14.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:14.643 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:15:14.644 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:14.644 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:14.644 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:14.645 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:14.645 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:14.645 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:14.645 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:14.646 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:14.646 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:14.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:14.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:14.648 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:15:14.648 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:14.648 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:14.648 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:14.648 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:14.649 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:14.649 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:14.649 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:14.649 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:14.650 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:14.650 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:14.650 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:15:14.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:14.650 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:15:14.651 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:15:14.651 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:15:14.651 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:14.651 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:15:14.652 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:15:19.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:19.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:19.655 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:15:19.727 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:15:19.727 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:15:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:15:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:15:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:15:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:15:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:15:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:15:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:15:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:15:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:15:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:15:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:15:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:15:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:15:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:15:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:15:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:15:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:15:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:15:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:15:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:15:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:19.800 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:15:19.802 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.089761_243.1 2021-09-17T02:15:24.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:24.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:24.805 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:15:24.849 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:15:24.849 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:15:24.901 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:15:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:15:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:15:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:15:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:15:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:15:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:15:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:15:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:15:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:15:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:15:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:15:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:15:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:15:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:15:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:15:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:15:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:15:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:15:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:15:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:24.916 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:15:24.917 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.048044_234.1 2021-09-17T02:15:29.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:29.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:29.922 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:15:29.971 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:15:29.972 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:15:30.023 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:15:30.024 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:15:30.024 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:15:30.024 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:15:30.024 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:15:30.025 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:15:30.025 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:30.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:30.025 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:15:30.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:30.026 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:15:30.026 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:30.026 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:30.026 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:15:30.027 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:30.027 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:15:30.027 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:30.027 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:15:30.027 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:30.028 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:30.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:30.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:30.028 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:15:30.029 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:30.029 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:30.029 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:15:30.029 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:30.029 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:30.030 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:30.030 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:15:30.030 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:30.030 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:30.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:30.031 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:30.031 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:15:30.031 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:30.032 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:30.032 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:30.032 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:30.032 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:30.033 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:30.033 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:30.033 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:30.033 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:30.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:30.034 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:30.034 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:15:30.034 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:30.034 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:30.035 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:30.035 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:30.035 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:30.035 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:30.036 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:30.036 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:30.036 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:30.036 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:30.036 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:15:30.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:30.037 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:15:30.037 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:15:30.037 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:15:30.037 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:30.038 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:15:30.039 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.065659_239.1 2021-09-17T02:15:35.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:35.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:35.042 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:15:35.089 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:15:35.089 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:15:35.144 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:15:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:15:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:15:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:15:35.145 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:15:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:15:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:35.146 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:15:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:15:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:35.147 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:15:35.148 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:15:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:15:35.149 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:35.150 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:15:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:15:35.151 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:15:35.152 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:35.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:15:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:35.154 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:35.155 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:35.156 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:15:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:35.157 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:35.158 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:35.159 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:15:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:15:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:15:35.160 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:15:35.161 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:35.161 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:15:35.162 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.067914_240.1 2021-09-17T02:15:40.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:40.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:40.165 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:15:40.213 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:15:40.214 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:15:40.268 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:15:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:15:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:15:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:15:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:15:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:15:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:15:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:15:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:15:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:15:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:15:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:15:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:15:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:15:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:15:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:15:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:15:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:40.284 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:15:40.284 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:15:40.284 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:15:40.284 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:40.284 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:15:40.286 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.064665_238.1 2021-09-17T02:15:45.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:45.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:45.289 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:15:45.340 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:15:45.341 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:15:45.400 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:15:45.400 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:15:45.400 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:15:45.401 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:15:45.401 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:15:45.401 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:15:45.401 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:45.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:45.402 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:15:45.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:45.402 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:15:45.402 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:45.403 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:45.403 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:15:45.403 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:45.403 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:15:45.404 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:45.404 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:15:45.404 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:45.404 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:45.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:45.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:45.405 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:15:45.405 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:45.406 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:45.406 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.103309_245.1", 2021-09-17T02:15:45.406 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:45.407 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:45.407 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:45.407 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.103309Z", 2021-09-17T02:15:45.407 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:45.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:45.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:45.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:45.408 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:15:45.408 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:45.409 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:45.409 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:45.409 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:45.409 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:45.409 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:45.410 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:45.410 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:45.410 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:45.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:45.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:45.411 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:15:45.411 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:45.411 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:45.411 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:45.412 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:45.412 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:45.412 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:45.412 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:45.413 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:45.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:45.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:45.414 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:15:45.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:45.414 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:15:45.414 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:15:45.415 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:15:45.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:45.416 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:15:45.417 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.103309_245.1 2021-09-17T02:15:50.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:50.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:50.422 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:15:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:15:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:15:50.531 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:15:50.531 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:15:50.532 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:15:50.532 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:15:50.532 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:15:50.532 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:15:50.532 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:50.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:50.533 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:15:50.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:50.533 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:15:50.534 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:50.534 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:50.534 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:15:50.534 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:50.534 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:15:50.535 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:50.535 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:15:50.535 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:50.535 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:50.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:50.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:50.536 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:15:50.536 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:50.536 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:50.537 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.290806_258.1", 2021-09-17T02:15:50.537 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:50.537 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:50.537 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:50.537 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.290806Z", 2021-09-17T02:15:50.538 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:50.538 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:50.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:50.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:50.539 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:15:50.539 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:50.539 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:50.539 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:50.539 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:50.540 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:50.540 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:50.540 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:50.540 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:50.540 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:50.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:50.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:50.541 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:15:50.541 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:50.542 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:50.542 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:50.542 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:50.542 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:50.542 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:50.543 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:50.543 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:50.543 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:50.543 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:50.543 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:15:50.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:50.544 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:15:50.544 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:15:50.544 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:15:50.545 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:50.545 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:15:50.546 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.290806_258.1 2021-09-17T02:15:55.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:55.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:15:55.549 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:15:55.600 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:15:55.600 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:15:55.661 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:15:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:15:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:15:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:15:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:15:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:15:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:15:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:15:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:15:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:15:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:15:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:15:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:15:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:15:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:15:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:15:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:15:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:15:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:15:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:15:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:15:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:15:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:15:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:15:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:15:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:15:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:15:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:15:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:15:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:15:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:15:55.675 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:15:55.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.065659_239.1 2021-09-17T02:16:00.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:00.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:00.679 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:00.727 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:00.783 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:00.783 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:00.784 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:00.784 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:00.784 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:00.784 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:00.784 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:00.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:00.785 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:00.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:00.786 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:00.786 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:00.786 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:00.786 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:00.786 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:00.787 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:00.787 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:00.787 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:00.787 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:00.788 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:00.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:00.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:00.788 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:00.788 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:00.789 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:00.789 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:16:00.789 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:00.789 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:00.789 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:00.790 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:16:00.790 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:00.790 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:00.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:00.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:00.791 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:00.791 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:00.791 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:00.791 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:00.792 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:00.792 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:00.792 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:00.792 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:00.793 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:00.793 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:00.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:00.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:00.793 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:00.794 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:00.794 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:00.794 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:00.794 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:00.795 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:00.795 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:00.795 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:00.795 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:00.795 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:00.796 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:00.796 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:00.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:00.796 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:00.796 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:00.797 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:00.797 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:00.797 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:00.798 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.048044_234.1 2021-09-17T02:16:05.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:05.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:05.801 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:06.322 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:06.323 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:06.385 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:06.386 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:06.386 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:06.386 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:06.387 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:06.387 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:06.387 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:06.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:06.387 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:06.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:06.388 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:06.388 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:06.388 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:06.389 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:06.389 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:06.389 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:06.389 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:06.389 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:06.390 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:06.390 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:06.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:06.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:06.391 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:06.391 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:06.391 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:06.391 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.051622_236.1", 2021-09-17T02:16:06.392 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:06.392 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:06.392 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:06.392 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.051622Z", 2021-09-17T02:16:06.392 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:06.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:06.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:06.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:06.393 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:06.393 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:06.394 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:06.394 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:06.394 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:06.394 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:06.395 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:06.395 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:06.395 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:06.395 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:06.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:06.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:06.396 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:06.396 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:06.396 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:06.396 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:06.397 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:06.397 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:06.397 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:06.397 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:06.398 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:06.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:06.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:06.398 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:06.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:06.399 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:06.399 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:06.399 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:06.399 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:06.399 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:06.400 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.051622_236.1 2021-09-17T02:16:11.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:11.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:11.403 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:11.455 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:11.456 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:11.515 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:11.515 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:11.516 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:11.516 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:11.516 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:11.517 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:11.517 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:11.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:11.517 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:11.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:11.518 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:11.518 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:11.518 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:11.518 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:11.519 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:11.519 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:11.519 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:11.519 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:11.519 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:11.520 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:11.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:11.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:11.520 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:11.521 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:11.521 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:11.521 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.161433_248.1", 2021-09-17T02:16:11.522 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:11.522 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:11.522 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:11.522 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.161433Z", 2021-09-17T02:16:11.522 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:11.523 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:11.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:11.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:11.523 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:11.524 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:11.524 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:11.524 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:11.524 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:11.525 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:11.525 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:11.525 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:11.525 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:11.525 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:11.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:11.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:11.526 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:11.526 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:11.527 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:11.527 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:11.527 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:11.527 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:11.527 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:11.528 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:11.528 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:11.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:11.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:11.529 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:11.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:11.529 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:11.529 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:11.529 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:11.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:11.530 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:11.531 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.161433_248.1 2021-09-17T02:16:16.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:16.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:16.534 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:16.586 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:16.587 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:16.649 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:16.650 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:16.650 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:16.650 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:16.651 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:16.651 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:16.651 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:16.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:16.651 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:16.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:16.652 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:16.652 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:16.652 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:16.653 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:16.653 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:16.653 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:16.653 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:16.653 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:16.654 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:16.654 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:16.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:16.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:16.655 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:16.655 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:16.655 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:16.655 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:16:16.655 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:16.656 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:16.656 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:16.656 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:16:16.656 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:16.657 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:16.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:16.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:16.657 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:16.657 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:16.658 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:16.658 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:16.658 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:16.658 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:16.659 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:16.659 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:16.659 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:16.659 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:16.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:16.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:16.660 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:16.660 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:16.660 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:16.661 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:16.661 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:16.661 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:16.661 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:16.661 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:16.662 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:16.662 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:16.662 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:16.662 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:16.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:16.663 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:16.663 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:16.663 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:16.663 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:16.664 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:16.665 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.065659_239.1 2021-09-17T02:16:21.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:21.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:21.668 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:21.737 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:21.797 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:21.798 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:21.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:21.800 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:21.801 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:21.802 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:21.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:21.804 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:16:21.805 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:21.806 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:16:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:21.807 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:21.808 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:21.809 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:21.810 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:21.811 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:21.812 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:21.813 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:21.814 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:21.815 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:21.816 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:21.816 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:21.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.269640_253.1 2021-09-17T02:16:26.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:26.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:26.821 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:26.872 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:26.873 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:26.934 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:26.935 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:26.935 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:26.935 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:26.936 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:26.936 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:26.936 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:26.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:26.936 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:26.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:26.937 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:26.937 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:26.937 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:26.938 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:26.938 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:26.938 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:26.938 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:26.938 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:26.939 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:26.939 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:26.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:26.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:26.940 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:26.940 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:26.940 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:26.940 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:16:26.940 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:26.941 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:26.941 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:26.941 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:16:26.941 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:26.941 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:26.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:26.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:26.942 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:26.942 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:26.943 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:26.943 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:26.943 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:26.943 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:26.944 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:26.944 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:26.944 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:26.944 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:26.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:26.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:26.945 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:26.945 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:26.945 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:26.945 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:26.946 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:26.946 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:26.946 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:26.946 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:26.947 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:26.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:26.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:26.947 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:26.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:26.948 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:26.948 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:26.948 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:26.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:26.949 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:26.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.067914_240.1 2021-09-17T02:16:31.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:31.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:31.952 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:32.002 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:32.003 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:32.063 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:32.063 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:32.064 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:32.064 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:32.064 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:32.064 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:32.065 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:32.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:32.065 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:32.066 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:32.066 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:32.066 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:32.066 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:32.067 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:32.067 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:32.067 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:32.067 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:32.068 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:32.068 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:32.068 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:32.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:32.069 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:32.069 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:32.069 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:32.069 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:32.069 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.212386_250.1", 2021-09-17T02:16:32.070 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:32.070 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:32.070 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:32.070 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.212386Z", 2021-09-17T02:16:32.071 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:32.071 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:32.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:32.071 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:32.071 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:32.072 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:32.072 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:32.072 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:32.072 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:32.073 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:32.073 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:32.073 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:32.073 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:32.074 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:32.074 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:32.074 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:32.074 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:32.074 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:32.075 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:32.075 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:32.075 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:32.075 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:32.076 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:32.076 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:32.076 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:32.076 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:32.077 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:32.078 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:32.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:32.078 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:32.078 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:32.079 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:32.079 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:32.079 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:32.080 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.212386_250.1 2021-09-17T02:16:37.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:37.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:37.083 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:37.136 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:37.137 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:37.198 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:37.198 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:37.199 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:37.199 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:37.199 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:37.199 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:37.199 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:37.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:37.200 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:37.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:37.200 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:37.201 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:37.201 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:37.201 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:37.201 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:37.201 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:37.202 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:37.202 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:37.202 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:37.202 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:37.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:37.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:37.203 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:37.203 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:37.203 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:37.204 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:16:37.204 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:37.204 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:37.204 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:37.204 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:16:37.205 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:37.205 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:37.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:37.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:37.205 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:37.206 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:37.206 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:37.206 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:37.206 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:37.206 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:37.207 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:37.207 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:37.207 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:37.207 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:37.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:37.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:37.208 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:37.208 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:37.208 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:37.208 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:37.209 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:37.209 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:37.209 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:37.209 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:37.210 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:37.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:37.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:37.210 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:37.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:37.211 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:37.211 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:37.211 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:37.211 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:37.211 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:37.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:16:42.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:42.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:42.216 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:42.268 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:42.269 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:42.329 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:42.330 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:42.330 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:42.330 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:42.330 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:42.331 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:42.331 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:42.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:42.331 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:42.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:42.332 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:42.332 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:42.332 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:42.332 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:42.333 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:42.333 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:42.333 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:42.333 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:42.333 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:42.334 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:42.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:42.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:42.334 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:42.335 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:42.335 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:42.335 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:16:42.335 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:42.335 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:42.336 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:42.336 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:16:42.336 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:42.336 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:42.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:42.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:42.337 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:42.337 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:42.337 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:42.338 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:42.338 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:42.338 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:42.338 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:42.338 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:42.339 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:42.339 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:42.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:42.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:42.340 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:42.340 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:42.340 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:42.341 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:42.341 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:42.341 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:42.341 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:42.341 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:42.342 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:42.342 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:42.342 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:42.342 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:42.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:42.343 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:42.343 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:42.343 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:42.343 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:42.344 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:42.346 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.065659_239.1 2021-09-17T02:16:47.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:47.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:47.348 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:47.397 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:47.398 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:47.458 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:47.459 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:47.459 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:47.459 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:47.459 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:47.460 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:47.460 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:47.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:47.460 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:47.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:47.461 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:47.461 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:47.461 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:47.461 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:47.461 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:47.462 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:47.462 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:47.462 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:47.462 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:47.463 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:47.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:47.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:47.463 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:47.463 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:47.464 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:47.464 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:16:47.464 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:47.464 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:47.464 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:47.465 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:16:47.465 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:47.465 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:47.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:47.466 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:47.466 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:47.466 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:47.466 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:47.466 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:47.467 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:47.468 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:47.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:47.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:47.469 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:47.469 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:47.469 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:47.470 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:47.470 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:47.470 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:47.470 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:47.470 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:47.471 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:47.471 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:47.471 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:47.471 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:47.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:47.472 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:47.472 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:47.472 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:47.472 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:47.473 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:47.474 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.065659_239.1 2021-09-17T02:16:52.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:52.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:52.476 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:52.532 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:52.532 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:52.597 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:52.597 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:52.598 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:52.598 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:52.598 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:52.598 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:52.599 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:52.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:52.599 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:52.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:52.600 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:52.600 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:52.600 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:52.600 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:52.600 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:52.601 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:52.601 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:52.601 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:52.601 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:52.601 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:52.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:52.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:52.602 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:52.602 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:52.603 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:52.603 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.272193_254.1", 2021-09-17T02:16:52.603 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:52.603 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:52.604 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:52.604 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.272193Z", 2021-09-17T02:16:52.604 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:52.604 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:52.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:52.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:52.605 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:52.605 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:52.605 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:52.606 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:52.606 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:52.606 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:52.606 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:52.606 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:52.607 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:52.607 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:52.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:52.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:52.607 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:52.608 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:52.608 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:52.608 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:52.608 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:52.609 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:52.609 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:52.609 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:52.609 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:52.609 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:52.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:52.610 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:52.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:52.610 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:52.610 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:52.611 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:52.611 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:52.611 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:52.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.272193_254.1 2021-09-17T02:16:57.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:57.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:16:57.616 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:16:57.668 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:16:57.669 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:16:57.731 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:16:57.731 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:16:57.732 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:16:57.732 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:16:57.732 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:16:57.732 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:16:57.733 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:57.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:57.733 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:16:57.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:57.734 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:16:57.734 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:57.734 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:57.734 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:16:57.734 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:57.735 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:16:57.735 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:57.735 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:16:57.735 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:57.736 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:57.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:57.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:57.736 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:16:57.736 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:57.737 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:57.737 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:16:57.737 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:57.737 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:57.738 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:57.738 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:16:57.738 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:57.738 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:57.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:57.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:57.739 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:16:57.739 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:57.739 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:57.740 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:57.740 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:57.740 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:57.740 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:57.741 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:57.741 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:57.741 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:57.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:57.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:16:57.742 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:16:57.742 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:16:57.742 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:16:57.742 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:16:57.743 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:16:57.743 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:16:57.743 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:16:57.743 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:16:57.744 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:16:57.744 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:57.744 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:57.744 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:16:57.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:16:57.745 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:16:57.745 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:16:57.745 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:16:57.745 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:16:57.745 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:16:57.747 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:17:02.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:02.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:02.749 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:02.801 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:02.801 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:17:02.863 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:17:02.863 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:17:02.863 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:17:02.864 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:17:02.864 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:17:02.864 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:17:02.865 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:02.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:02.865 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:17:02.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:02.866 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:17:02.866 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:02.866 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:02.866 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:17:02.867 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:02.867 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:17:02.867 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:02.867 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:17:02.868 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:02.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:02.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:02.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:02.868 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:17:02.869 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:02.869 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:02.869 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.051622_236.1", 2021-09-17T02:17:02.869 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:02.870 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:02.870 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:02.871 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.051622Z", 2021-09-17T02:17:02.872 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:02.872 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:02.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:02.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:02.872 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:17:02.873 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:02.873 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:02.873 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:02.873 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:02.874 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:02.874 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:02.874 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:02.874 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:02.874 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:02.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:02.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:02.875 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:17:02.875 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:02.876 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:02.876 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:02.876 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:02.876 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:02.877 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:02.877 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:02.877 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:02.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:02.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:02.878 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:17:02.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:02.878 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:17:02.878 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:17:02.879 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:17:02.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:02.879 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:17:02.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.051622_236.1 2021-09-17T02:17:07.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:07.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:07.883 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:07.932 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:07.933 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:17:07.993 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:17:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:17:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:17:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:17:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:17:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:17:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:17:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:17:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:17:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:17:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:17:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:17:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:17:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:17:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:17:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:17:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:17:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:17:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:17:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:17:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:08.007 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:17:08.008 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.048044_234.1 2021-09-17T02:17:13.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:13.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:13.011 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:13.062 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:13.063 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:17:13.122 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:17:13.123 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:17:13.123 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:17:13.123 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:17:13.123 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:17:13.123 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:17:13.124 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:13.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:13.124 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:17:13.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:13.125 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:17:13.125 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:13.125 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:13.125 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:17:13.125 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:13.126 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:17:13.126 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:13.126 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:17:13.126 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:13.127 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:13.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:13.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:13.127 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:17:13.127 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:13.128 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:13.128 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:17:13.128 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:13.128 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:13.129 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:13.129 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:17:13.129 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:13.129 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:13.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:13.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:13.130 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:17:13.130 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:13.130 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:13.130 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:13.131 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:13.131 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:13.131 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:13.131 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:13.132 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:13.132 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:13.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:13.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:13.132 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:17:13.133 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:13.133 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:13.133 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:13.133 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:13.133 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:13.134 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:13.134 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:13.134 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:13.134 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:13.135 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:13.135 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:17:13.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:13.135 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:17:13.135 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:17:13.136 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:17:13.136 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:13.136 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:17:13.137 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.064665_238.1 2021-09-17T02:17:18.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:18.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:18.140 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:18.191 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:18.191 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:17:18.289 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:17:18.290 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:17:18.290 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:17:18.290 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:17:18.291 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:17:18.291 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:17:18.291 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:18.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:18.292 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:17:18.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:18.292 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:17:18.292 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:18.292 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:18.293 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:17:18.293 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:18.293 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:17:18.293 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:18.294 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:17:18.294 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:18.294 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:18.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:18.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:18.295 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:17:18.295 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:18.295 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:18.295 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:17:18.295 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:18.296 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:18.296 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:18.296 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:17:18.296 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:18.296 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:18.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:18.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:18.297 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:17:18.297 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:18.297 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:18.298 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:18.298 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:18.298 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:18.298 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:18.299 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:18.299 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:18.299 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:18.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:18.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:18.300 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:17:18.300 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:18.300 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:18.300 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:18.300 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:18.301 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:18.301 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:18.301 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:18.301 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:18.301 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:18.302 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:18.302 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:17:18.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:18.302 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:17:18.303 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:17:18.303 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:17:18.303 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:18.303 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:17:18.304 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:17:23.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:23.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:23.307 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:23.381 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:23.382 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:17:23.786 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:17:23.787 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:17:23.787 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:17:23.787 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:17:23.787 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:17:23.787 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:17:23.788 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:23.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:23.788 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:17:23.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:23.789 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:17:23.789 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:23.789 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:23.789 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:17:23.789 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:23.790 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:17:23.790 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:23.790 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:17:23.790 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:23.791 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:23.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:23.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:23.791 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:17:23.791 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:23.792 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:23.792 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:17:23.792 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:23.792 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:23.792 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:23.793 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:17:23.793 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:23.793 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:23.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:23.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:23.794 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:17:23.794 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:23.794 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:23.794 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:23.795 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:23.795 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:23.795 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:23.795 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:23.795 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:23.796 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:23.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:23.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:23.796 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:17:23.796 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:23.797 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:23.797 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:23.797 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:23.797 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:23.798 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:23.798 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:23.798 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:23.798 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:23.798 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:23.799 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:17:23.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:23.799 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:17:23.799 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:17:23.799 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:17:23.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:23.800 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:17:23.801 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.048044_234.1 2021-09-17T02:17:28.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:28.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:28.803 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:28.872 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:28.872 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:17:28.939 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:17:28.939 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:17:28.940 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:17:28.940 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:17:28.940 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:17:28.940 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:17:28.940 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:28.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:28.941 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:17:28.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:28.941 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:17:28.942 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:28.942 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:28.942 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:17:28.942 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:28.943 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:17:28.943 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:28.943 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:17:28.943 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:28.943 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:28.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:28.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:28.944 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:17:28.944 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:28.945 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:28.945 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.051622_236.1", 2021-09-17T02:17:28.945 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:28.945 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:28.945 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:28.946 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.051622Z", 2021-09-17T02:17:28.946 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:28.946 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:28.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:28.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:28.947 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:17:28.947 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:28.947 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:28.947 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:28.948 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:28.948 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:28.948 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:28.948 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:28.948 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:28.949 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:28.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:28.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:28.949 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:17:28.950 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:28.950 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:28.950 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:28.950 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:28.951 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:28.951 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:28.951 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:28.951 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:28.951 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:28.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:28.952 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:17:28.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:28.952 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:17:28.952 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:17:28.953 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:17:28.953 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:28.953 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:17:28.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.051622_236.1 2021-09-17T02:17:33.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:33.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:33.957 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:34.036 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:34.036 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:17:34.143 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:17:34.144 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:17:34.144 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:17:34.144 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:17:34.144 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:17:34.145 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:17:34.145 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:34.145 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:34.145 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:17:34.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:34.146 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:17:34.146 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:34.147 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:34.148 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:17:34.148 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:34.149 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:17:34.149 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:34.149 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:17:34.150 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:34.150 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:34.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:34.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:34.152 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:17:34.152 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:34.153 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:34.153 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:17:34.153 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:34.154 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:34.154 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:34.155 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:17:34.155 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:34.156 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:34.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:34.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:34.157 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:17:34.158 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:34.158 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:34.159 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:34.159 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:34.160 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:34.160 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:34.160 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:34.161 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:34.161 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:34.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:34.162 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:34.163 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:17:34.163 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:34.164 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:34.164 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:34.165 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:34.165 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:34.165 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:34.166 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:34.166 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:34.167 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:34.167 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:34.168 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:17:34.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:34.168 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:17:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:17:34.169 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:17:34.170 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:34.170 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:17:34.172 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.067914_240.1 2021-09-17T02:17:39.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:39.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:39.176 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:39.223 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:39.223 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:17:39.318 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:17:39.319 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:17:39.319 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:17:39.319 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:17:39.319 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:17:39.319 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:17:39.320 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:39.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:39.320 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:17:39.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:39.320 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:17:39.321 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:39.321 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:39.321 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:17:39.321 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:39.322 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:17:39.322 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:39.322 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:17:39.322 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:39.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:39.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:39.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:39.323 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:17:39.323 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:39.324 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:39.324 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:17:39.324 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:39.324 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:39.324 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:39.325 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:17:39.325 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:39.325 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:39.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:39.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:39.326 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:17:39.326 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:39.326 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:39.326 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:39.327 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:39.327 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:39.327 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:39.328 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:39.328 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:39.328 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:39.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:39.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:39.329 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:17:39.329 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:39.329 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:39.329 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:39.329 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:39.330 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:39.330 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:39.330 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:39.330 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:39.331 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:39.331 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:39.331 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:17:39.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:39.331 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:17:39.332 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:17:39.332 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:17:39.332 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:39.332 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:17:39.334 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.065659_239.1 2021-09-17T02:17:44.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:44.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:44.336 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:44.399 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:44.399 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:17:44.490 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:17:44.495 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:17:44.495 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:17:44.495 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:17:44.496 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:17:44.496 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:17:44.496 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:44.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:44.497 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:17:44.497 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:44.497 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:17:44.497 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:44.498 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:44.498 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:17:44.498 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:44.498 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:17:44.499 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:44.499 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:17:44.499 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:44.499 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:44.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:44.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:44.500 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:17:44.500 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:44.501 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:44.501 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:17:44.501 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:44.501 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:44.501 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:44.502 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:17:44.502 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:44.502 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:44.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:44.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:44.503 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:17:44.503 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:44.503 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:44.503 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:44.504 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:44.504 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:44.504 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:44.504 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:44.505 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:44.505 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:44.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:44.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:44.506 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:17:44.506 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:44.506 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:44.506 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:44.506 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:44.507 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:44.507 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:44.507 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:44.507 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:44.508 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:44.508 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:44.508 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:17:44.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:44.508 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:17:44.509 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:17:44.509 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:17:44.509 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:44.509 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:17:44.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.089761_243.1 2021-09-17T02:17:49.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:49.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:49.513 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:49.583 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:49.584 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:17:49.669 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:17:49.670 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:17:49.670 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:17:49.670 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:17:49.670 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:17:49.671 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:17:49.671 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:49.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:49.671 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:17:49.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:49.672 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:17:49.672 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:49.672 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:49.672 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:17:49.673 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:49.673 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:17:49.673 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:49.673 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:17:49.674 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:49.674 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:49.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:49.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:49.674 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:17:49.675 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:49.675 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:49.675 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:17:49.675 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:49.676 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:49.676 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:49.676 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:17:49.676 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:49.676 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:49.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:49.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:49.677 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:17:49.677 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:49.677 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:49.678 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:49.678 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:49.678 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:49.678 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:49.678 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:49.679 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:49.679 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:49.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:49.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:49.680 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:17:49.680 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:49.680 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:49.680 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:49.680 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:49.681 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:49.681 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:49.681 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:49.681 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:49.681 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:49.682 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:49.682 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:17:49.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:49.682 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:17:49.683 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:17:49.683 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:17:49.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:49.683 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:17:49.684 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:17:54.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:54.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:54.687 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:54.760 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:54.761 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:17:54.891 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:17:54.891 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:17:54.891 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:17:54.892 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:17:54.892 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:17:54.892 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:17:54.892 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:54.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:54.893 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:17:54.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:54.893 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:17:54.893 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:54.894 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:54.894 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:17:54.894 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:54.894 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:17:54.894 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:54.895 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:17:54.895 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:54.895 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:54.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:54.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:54.896 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:17:54.896 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:54.896 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:54.896 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.288419_257.1", 2021-09-17T02:17:54.897 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:54.897 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:54.897 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:54.897 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.288419Z", 2021-09-17T02:17:54.897 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:54.898 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:54.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:54.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:54.898 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:17:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:54.899 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:54.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:17:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:17:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:17:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:17:54.901 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:17:54.902 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:17:54.902 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:17:54.902 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:17:54.902 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:17:54.903 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:17:54.903 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:54.903 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:54.903 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:17:54.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:17:54.904 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:17:54.904 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:17:54.904 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:17:54.904 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:17:54.904 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:17:54.906 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.288419_257.1 2021-09-17T02:17:59.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:59.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:17:59.908 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:17:59.978 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:17:59.979 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:00.337 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:00.338 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:00.338 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:00.338 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:00.338 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:00.339 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:00.339 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:00.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:00.339 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:00.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:00.340 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:00.340 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:00.340 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:00.340 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:00.341 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:00.341 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:00.341 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:00.341 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:00.341 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:00.342 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:00.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:00.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:00.342 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:00.343 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:00.343 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:00.343 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:18:00.344 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:00.344 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:00.344 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:00.344 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:18:00.345 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:00.345 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:00.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:00.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:00.346 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:00.347 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:00.347 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:00.348 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:00.348 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:00.348 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:00.348 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:00.349 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:00.349 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:00.349 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:00.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:00.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:00.350 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:00.351 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:00.351 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:00.351 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:00.352 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:00.352 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:00.352 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:00.353 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:00.353 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:00.353 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:00.353 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:00.354 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:00.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:00.354 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:00.355 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:00.355 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:00.355 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:00.356 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:00.358 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:18:05.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:05.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:05.360 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:18:05.437 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:18:05.438 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:05.528 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:05.529 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:05.530 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:05.531 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:05.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:05.533 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:05.534 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:05.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:05.536 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:05.537 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:05.538 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:05.538 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:05.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:05.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:05.539 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:05.539 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:05.539 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:05.539 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:05.539 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:05.540 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:05.540 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:05.540 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:05.540 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:05.540 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:05.541 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:05.541 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:05.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:05.541 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:05.541 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:05.542 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:05.542 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:05.542 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:05.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.089761_243.1 2021-09-17T02:18:10.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:10.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:10.548 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:18:10.594 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:18:10.595 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:10.652 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:10.652 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:10.652 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:10.652 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:10.653 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:10.653 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:10.653 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:10.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:10.654 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:10.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:10.654 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:10.654 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:10.655 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:10.655 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:10.655 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:10.655 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:10.655 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:10.656 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:10.656 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:10.656 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:10.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:10.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:10.657 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:10.657 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:10.657 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:10.657 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:18:10.658 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:10.658 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:10.658 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:10.658 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:18:10.659 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:10.659 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:10.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:10.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:10.659 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:10.660 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:10.660 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:10.660 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:10.660 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:10.660 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:10.661 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:10.661 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:10.661 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:10.661 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:10.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:10.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:10.662 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:10.662 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:10.662 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:10.663 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:10.663 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:10.663 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:10.663 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:10.663 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:10.664 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:10.664 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:10.664 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:10.664 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:10.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:10.665 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:10.665 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:10.665 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:10.665 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:10.665 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:10.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.067914_240.1 2021-09-17T02:18:15.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:15.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:15.669 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:18:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:18:15.723 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:15.778 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:15.778 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:15.779 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:15.779 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:15.779 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:15.780 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:15.780 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:15.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:15.780 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:15.781 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:15.781 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:15.781 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:15.781 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:15.781 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:15.781 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:15.782 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:15.782 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:15.782 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:15.782 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:15.783 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:15.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:15.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:15.783 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:15.783 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:15.784 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:15.784 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:18:15.784 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:15.784 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:15.785 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:15.785 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:18:15.785 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:15.785 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:15.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:15.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:15.786 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:15.786 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:15.786 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:15.787 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:15.787 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:15.787 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:15.787 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:15.787 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:15.788 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:15.788 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:15.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:15.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:15.789 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:15.790 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:15.790 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:15.790 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:15.791 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:15.791 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:15.791 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:15.791 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:15.794 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:15.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.067914_240.1 2021-09-17T02:18:20.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:20.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:20.798 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:18:20.843 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:18:20.844 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:20.897 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:20.897 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:20.898 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:20.898 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:20.898 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:20.898 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:20.899 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:20.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:20.899 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:20.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:20.899 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:20.900 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:20.900 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:20.900 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:20.900 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:20.900 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:20.901 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:20.901 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:20.901 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:20.901 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:20.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:20.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:20.902 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:20.902 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:20.902 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:20.903 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.103309_245.1", 2021-09-17T02:18:20.903 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:20.903 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:20.903 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:20.904 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.103309Z", 2021-09-17T02:18:20.904 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:20.904 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:20.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:20.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:20.905 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:20.905 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:20.905 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:20.905 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:20.905 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:20.906 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:20.906 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:20.906 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:20.906 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:20.907 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:20.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:20.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:20.907 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:20.907 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:20.908 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:20.908 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:20.908 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:20.908 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:20.909 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:20.909 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:20.909 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:20.909 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:20.909 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:20.910 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:20.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:20.910 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:20.910 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:20.910 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:20.911 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:20.911 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:20.912 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.103309_245.1 2021-09-17T02:18:25.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:25.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:25.914 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:18:25.959 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:18:25.960 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:26.017 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:26.017 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:26.018 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:26.019 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:26.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:26.019 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:26.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:26.020 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:26.020 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:26.020 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:26.020 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:26.020 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:26.021 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:26.021 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:26.021 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:26.021 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:26.022 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:26.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:26.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:26.022 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:26.022 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:26.023 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:26.023 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:18:26.023 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:26.023 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:26.024 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:26.024 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:18:26.024 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:26.024 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:26.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:26.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:26.025 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:26.025 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:26.025 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:26.025 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:26.026 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:26.026 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:26.026 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:26.026 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:26.027 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:26.027 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:26.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:26.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:26.027 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:26.028 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:26.028 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:26.028 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:26.028 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:26.028 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:26.029 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:26.029 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:26.029 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:26.029 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:26.030 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:26.030 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:26.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:26.030 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:26.030 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:26.031 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:26.031 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:26.031 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:26.032 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.067914_240.1 2021-09-17T02:18:31.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:31.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:31.035 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:18:31.078 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:18:31.078 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:31.131 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:31.132 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:31.132 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:31.132 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:31.132 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:31.133 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:31.133 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:31.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:31.133 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:31.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:31.134 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:31.134 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:31.134 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:31.134 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:31.135 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:31.135 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:31.135 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:31.135 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:31.135 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:31.136 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:31.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:31.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:31.136 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:31.136 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:31.137 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:31.137 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.062079_237.1", 2021-09-17T02:18:31.137 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:31.137 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:31.138 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:31.138 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.062079Z", 2021-09-17T02:18:31.138 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:31.138 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:31.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:31.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:31.141 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:31.141 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:31.141 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:31.142 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:31.142 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:31.143 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:31.143 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:31.143 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:31.143 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:31.143 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:31.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:31.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:31.144 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:31.144 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:31.145 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:31.145 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:31.145 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:31.145 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:31.145 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:31.146 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:31.146 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:31.146 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:31.146 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:31.147 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:31.147 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:31.147 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:31.147 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:31.147 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:31.148 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:31.148 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:31.149 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.062079_237.1 2021-09-17T02:18:36.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:36.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:36.153 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:18:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:18:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:36.256 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:36.257 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:36.257 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:36.257 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:36.258 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:36.258 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:36.258 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:36.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:36.259 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:36.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:36.259 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:36.259 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:36.259 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:36.260 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:36.260 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:36.260 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:36.261 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:36.261 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:36.261 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:36.261 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:36.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:36.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:36.262 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:36.262 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:36.262 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:36.263 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.051622_236.1", 2021-09-17T02:18:36.263 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:36.263 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:36.263 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:36.263 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.051622Z", 2021-09-17T02:18:36.264 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:36.264 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:36.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:36.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:36.264 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:36.265 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:36.265 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:36.265 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:36.265 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:36.265 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:36.266 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:36.266 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:36.266 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:36.267 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:36.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:36.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:36.268 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:36.268 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:36.268 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:36.268 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:36.269 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:36.269 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:36.269 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:36.269 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:36.269 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:36.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:36.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:36.270 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:36.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:36.271 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:36.271 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:36.271 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:36.271 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:36.271 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:36.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.051622_236.1 2021-09-17T02:18:41.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:41.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:41.275 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:18:41.358 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:18:41.359 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:41.418 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:41.419 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:41.419 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:41.419 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:41.420 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:41.420 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:41.420 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:41.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:41.420 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:41.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:41.421 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:41.421 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:41.421 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:41.422 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:41.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:41.424 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.051622_236.1", 2021-09-17T02:18:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:41.425 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.051622Z", 2021-09-17T02:18:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:41.426 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:41.428 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:41.429 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:41.430 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:41.431 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:41.432 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:41.433 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:41.434 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:41.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:41.435 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:41.436 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.051622_236.1 2021-09-17T02:18:46.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:46.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:46.438 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:18:46.484 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:18:46.484 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:46.538 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:46.538 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:46.539 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:46.540 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:46.541 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:46.542 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:46.543 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:18:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:46.544 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:18:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:46.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:46.546 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:46.547 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:46.548 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:46.549 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:46.550 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:46.551 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:46.552 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:46.552 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:46.553 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.089761_243.1 2021-09-17T02:18:51.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:51.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:51.556 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:18:51.600 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:18:51.600 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:51.655 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:51.655 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:51.656 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:51.657 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:51.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:51.657 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:51.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:51.657 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:51.658 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:51.658 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:51.658 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:51.658 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:51.659 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:51.659 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:51.659 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:51.659 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:51.659 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:51.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:51.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:51.660 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:51.660 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:51.661 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:51.661 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:18:51.661 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:51.661 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:51.661 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:51.662 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:18:51.662 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:51.662 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:51.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:51.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:51.663 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:51.663 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:51.663 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:51.663 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:51.664 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:51.664 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:51.664 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:51.664 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:51.664 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:51.664 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:51.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:51.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:51.665 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:51.665 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:51.666 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:51.666 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:51.666 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:51.666 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:51.667 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:51.667 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:51.667 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:51.667 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:51.667 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:51.668 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:51.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:51.668 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:51.668 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:51.668 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:51.669 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:51.669 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:51.670 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:18:56.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:56.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:18:56.673 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:18:56.717 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:18:56.717 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:18:56.770 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:18:56.770 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:18:56.771 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:18:56.771 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:18:56.771 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:18:56.771 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:18:56.771 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:56.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:56.772 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:18:56.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:56.772 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:18:56.773 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:56.773 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:56.773 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:18:56.773 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:56.773 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:18:56.774 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:56.774 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:18:56.774 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:56.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:56.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:56.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:56.775 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:18:56.775 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:56.775 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:56.776 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:18:56.776 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:56.776 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:56.776 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:56.777 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:18:56.777 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:56.777 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:56.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:56.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:56.778 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:18:56.778 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:56.778 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:56.778 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:56.778 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:56.779 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:56.779 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:56.779 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:56.779 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:56.779 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:56.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:56.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:18:56.780 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:18:56.780 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:18:56.780 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:18:56.781 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:18:56.781 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:18:56.781 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:18:56.781 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:18:56.782 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:18:56.782 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:18:56.782 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:56.782 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:56.782 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:18:56.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:18:56.783 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:18:56.783 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:18:56.783 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:18:56.783 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:18:56.784 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:18:56.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:19:01.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:01.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:01.788 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:19:01.835 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:01.835 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:01.890 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:19:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:19:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:19:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:19:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:19:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:19:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:19:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:19:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:19:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:19:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.062079_237.1", 2021-09-17T02:19:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.062079Z", 2021-09-17T02:19:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:19:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:19:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:19:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:19:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:19:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:01.904 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:01.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.062079_237.1 2021-09-17T02:19:06.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:06.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:06.908 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:19:06.953 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:06.954 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:07.007 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:07.007 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:19:07.007 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:19:07.008 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:19:07.008 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:19:07.008 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:07.008 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:07.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:07.009 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:19:07.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:07.009 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:19:07.010 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:07.010 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:07.010 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:19:07.010 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:07.011 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:19:07.011 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:07.011 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:19:07.011 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:07.011 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:07.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:07.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:07.012 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:19:07.012 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:07.012 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:07.013 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:19:07.013 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:07.013 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:07.013 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:07.013 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:19:07.014 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:07.014 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:07.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:07.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:07.015 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:19:07.015 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:07.015 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:07.015 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:07.016 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:07.016 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:07.016 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:07.016 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:07.017 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:07.017 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:07.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:07.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:07.017 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:19:07.018 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:07.018 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:07.018 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:07.018 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:07.019 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:07.019 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:07.019 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:07.019 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:07.019 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:07.020 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:07.020 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:07.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:07.020 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:19:07.021 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:19:07.021 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:19:07.021 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:07.021 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:07.022 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:19:12.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:12.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:12.026 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:19:12.072 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:12.073 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:12.125 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:12.125 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:19:12.126 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:19:12.126 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:19:12.126 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:19:12.127 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:12.127 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:12.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:12.127 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:19:12.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:12.128 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:19:12.128 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:12.128 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:12.128 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:19:12.129 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:12.129 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:19:12.129 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:12.129 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:19:12.130 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:12.130 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:12.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:12.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:12.131 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:19:12.131 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:12.131 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:12.131 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:19:12.132 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:12.132 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:12.132 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:12.132 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:19:12.133 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:12.133 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:12.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:12.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:12.134 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:19:12.134 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:12.134 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:12.134 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:12.135 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:12.135 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:12.135 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:12.135 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:12.136 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:12.136 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:12.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:12.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:12.137 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:19:12.138 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:12.138 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:12.138 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:12.138 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:12.139 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:12.140 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:12.140 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:12.140 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:12.140 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:12.141 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:12.141 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:12.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:12.141 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:19:12.142 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:19:12.142 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:19:12.142 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:12.142 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:12.143 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.048044_234.1 2021-09-17T02:19:17.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:17.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:17.146 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:19:17.205 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:17.206 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:17.258 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:19:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:19:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:19:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:19:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:19:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:19:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:19:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:19:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:19:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:19:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.048044_234.1", 2021-09-17T02:19:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.048044Z", 2021-09-17T02:19:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:19:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:19:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:19:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:19:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:19:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:17.272 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:17.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.048044_234.1 2021-09-17T02:19:22.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:22.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:22.275 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:19:22.619 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:22.619 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:22.679 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:22.679 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:19:22.679 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:19:22.680 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:19:22.680 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:19:22.680 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:22.680 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:22.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:22.681 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:19:22.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:22.681 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:19:22.681 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:22.682 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:22.682 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:19:22.682 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:22.682 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:19:22.682 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:22.683 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:19:22.683 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:22.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:22.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:22.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:22.684 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:19:22.684 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:22.684 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:22.684 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:19:22.685 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:22.685 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:22.685 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:22.685 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:19:22.686 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:22.686 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:22.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:22.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:22.686 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:19:22.687 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:22.687 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:22.687 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:22.687 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:22.688 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:22.688 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:22.688 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:22.688 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:22.688 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:22.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:22.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:22.689 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:19:22.689 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:22.689 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:22.690 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:22.690 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:22.690 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:22.690 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:22.691 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:22.691 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:22.691 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:22.691 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:22.691 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:22.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:22.692 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:19:22.692 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:19:22.692 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:19:22.692 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:22.693 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:22.694 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.065659_239.1 2021-09-17T02:19:27.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:27.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:27.697 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:19:27.750 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:27.751 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:27.803 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:27.804 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:19:27.804 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:19:27.804 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:19:27.804 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:19:27.805 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:27.805 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:27.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:27.805 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:19:27.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:27.806 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:19:27.806 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:27.806 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:27.806 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:19:27.807 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:27.807 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:19:27.807 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:27.807 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:19:27.807 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:27.808 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:27.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:27.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:27.808 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:19:27.809 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:27.809 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:27.809 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.065659_239.1", 2021-09-17T02:19:27.809 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:27.809 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:27.810 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:27.810 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.065659Z", 2021-09-17T02:19:27.810 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:27.810 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:27.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:27.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:27.811 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:19:27.811 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:27.811 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:27.812 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:27.812 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:27.812 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:27.812 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:27.812 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:27.813 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:27.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:27.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:27.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:27.813 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:19:27.814 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:27.814 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:27.814 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:27.814 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:27.815 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:27.815 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:27.815 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:27.815 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:27.815 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:27.816 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:27.816 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:27.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:27.816 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:19:27.816 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:19:27.817 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:19:27.817 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:27.817 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:27.818 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.065659_239.1 2021-09-17T02:19:32.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:32.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:32.821 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:19:32.866 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:32.867 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:32.921 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:19:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:19:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:19:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:19:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:19:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:19:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:19:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:19:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:19:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:19:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:32.927 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:19:32.927 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:32.927 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:32.927 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:32.927 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:19:32.928 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:32.928 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:32.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:32.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:32.928 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:19:32.929 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:32.929 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:32.929 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:32.929 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:32.929 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:19:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:19:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:19:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:19:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:32.935 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:32.936 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:19:37.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:37.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:37.938 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:19:37.983 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:37.984 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:38.037 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:38.037 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:19:38.037 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:19:38.038 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:19:38.038 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:19:38.038 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:38.038 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:38.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:38.039 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:19:38.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:38.039 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:19:38.040 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:38.040 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:38.040 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:19:38.040 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:38.040 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:19:38.041 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:38.041 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:19:38.041 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:38.041 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:38.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:38.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:38.042 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:19:38.042 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:38.042 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:38.043 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.157882_247.1", 2021-09-17T02:19:38.043 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:38.043 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:38.043 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:38.044 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.157882Z", 2021-09-17T02:19:38.044 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:38.044 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:38.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:38.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:38.045 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:19:38.045 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:38.045 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:38.045 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:38.046 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:38.046 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:38.046 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:38.046 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:38.047 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:38.047 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:38.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:38.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:38.047 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:19:38.048 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:38.048 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:38.048 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:38.048 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:38.048 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:38.049 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:38.049 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:38.049 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:38.049 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:38.050 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:38.050 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:38.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:38.050 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:19:38.051 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:19:38.051 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:19:38.051 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:38.051 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:38.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.157882_247.1 2021-09-17T02:19:43.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:43.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:43.056 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:19:43.102 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:43.102 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:43.158 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:43.158 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:19:43.159 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:19:43.159 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:19:43.159 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:19:43.159 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:43.160 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:43.160 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:43.160 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:19:43.160 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:43.161 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:19:43.161 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:43.161 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:43.161 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:19:43.161 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:43.162 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:19:43.162 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:43.162 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:19:43.162 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:43.163 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:43.163 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:43.163 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:43.163 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:19:43.163 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:43.164 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:43.164 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.049023_235.1", 2021-09-17T02:19:43.164 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:43.164 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:43.165 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:43.165 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.049023Z", 2021-09-17T02:19:43.165 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:43.165 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:43.165 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:43.166 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:43.166 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:19:43.166 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:43.166 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:43.166 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:43.167 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:43.167 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:43.167 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:43.167 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:43.168 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:43.168 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:43.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:43.168 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:43.168 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:19:43.169 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:43.169 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:43.169 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:43.169 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:43.169 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:43.170 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:43.170 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:43.170 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:43.170 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:43.171 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:43.171 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:43.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:43.171 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:19:43.171 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:19:43.172 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:19:43.172 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:43.172 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:43.173 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.049023_235.1 2021-09-17T02:19:48.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:48.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:48.176 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:19:48.227 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:48.227 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:48.288 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:19:48.289 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:19:48.289 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:19:48.289 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:19:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:19:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:19:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:48.293 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:48.293 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:19:48.293 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:48.294 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:19:48.294 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:48.295 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:19:48.295 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:48.295 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:19:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:19:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:19:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:19:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:48.304 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:48.304 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:48.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:48.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:19:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:48.306 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:48.306 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:48.306 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:48.306 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:48.306 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:19:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:19:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:19:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:48.308 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:48.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.086132_241.1 2021-09-17T02:19:53.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:53.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:19:53.312 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:19:53.359 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:53.415 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:53.415 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:19:53.416 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:19:53.416 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:19:53.416 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:19:53.417 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:53.417 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:53.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:53.417 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:19:53.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:53.418 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:19:53.418 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:53.418 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:53.419 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:19:53.419 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:53.419 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:19:53.419 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:53.420 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:19:53.420 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:53.420 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:53.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:53.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:53.421 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:19:53.421 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:53.421 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:53.422 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:19:53.422 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:53.422 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:53.422 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:53.423 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:19:53.423 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:53.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:53.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:53.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:53.424 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:19:53.424 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:53.424 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:53.425 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:53.425 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:53.425 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:53.425 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:53.426 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:53.426 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:53.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:53.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:53.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:53.427 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:19:53.427 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:53.427 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:19:53.428 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:19:53.428 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:19:53.428 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:19:53.428 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:19:53.429 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:19:53.429 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:53.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:53.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:53.429 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:53.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:53.430 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:19:53.430 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:19:53.430 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:19:53.431 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:53.431 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:53.432 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631844887.995835_351.1 target=1_1631844259.088776_242.1 2021-09-17T02:19:58.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2021-09-17T02:19:58.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:19:58.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:19:58.437 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T02:19:58.484 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:58.485 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:58.508 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:19:58.505+0000 7f1ee2713380 0 failed reading obj info from .rgw.root:zone_info.505278ea-3c6d-4c68-8442-578099047a49: (2) No such file or directory 2021-09-17T02:19:58.509 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:19:58.505+0000 7f1ee2713380 0 WARNING: could not read zone params for zone id=505278ea-3c6d-4c68-8442-578099047a49 name=test-zone3 2021-09-17T02:19:58.509 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:19:58.506+0000 7f1ee2713380 0 failed reading obj info from .rgw.root:zone_info.95b0d488-6c59-4a37-b3ed-d5fa65f27678: (2) No such file or directory 2021-09-17T02:19:58.509 INFO:teuthology.orchestra.run.smithi045.stderr:2021-09-17T02:19:58.506+0000 7f1ee2713380 0 WARNING: could not read zone params for zone id=95b0d488-6c59-4a37-b3ed-d5fa65f27678 name=test-zone1 2021-09-17T02:19:58.513 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:58.513 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T02:19:58.513 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone2", 2021-09-17T02:19:58.513 INFO:teuthology.orchestra.run.smithi045.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-17T02:19:58.513 INFO:teuthology.orchestra.run.smithi045.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-17T02:19:58.514 INFO:teuthology.orchestra.run.smithi045.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-17T02:19:58.514 INFO:teuthology.orchestra.run.smithi045.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-17T02:19:58.514 INFO:teuthology.orchestra.run.smithi045.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-17T02:19:58.514 INFO:teuthology.orchestra.run.smithi045.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-17T02:19:58.515 INFO:teuthology.orchestra.run.smithi045.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-17T02:19:58.516 INFO:teuthology.orchestra.run.smithi045.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-17T02:19:58.516 INFO:teuthology.orchestra.run.smithi045.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-17T02:19:58.516 INFO:teuthology.orchestra.run.smithi045.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-17T02:19:58.516 INFO:teuthology.orchestra.run.smithi045.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-17T02:19:58.517 INFO:teuthology.orchestra.run.smithi045.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-17T02:19:58.517 INFO:teuthology.orchestra.run.smithi045.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-17T02:19:58.517 INFO:teuthology.orchestra.run.smithi045.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-17T02:19:58.518 INFO:teuthology.orchestra.run.smithi045.stdout: "system_key": { 2021-09-17T02:19:58.518 INFO:teuthology.orchestra.run.smithi045.stdout: "access_key": "HXCQHMJWFX35AU7N", 2021-09-17T02:19:58.518 INFO:teuthology.orchestra.run.smithi045.stdout: "secret_key": "B32eJkOEzpLbQQLCICRnQRj2OnvubCoS" 2021-09-17T02:19:58.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:58.519 INFO:teuthology.orchestra.run.smithi045.stdout: "placement_pools": [ 2021-09-17T02:19:58.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:58.519 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "default-placement", 2021-09-17T02:19:58.520 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:19:58.520 INFO:teuthology.orchestra.run.smithi045.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-17T02:19:58.520 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_classes": { 2021-09-17T02:19:58.520 INFO:teuthology.orchestra.run.smithi045.stdout: "STANDARD": { 2021-09-17T02:19:58.520 INFO:teuthology.orchestra.run.smithi045.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-17T02:19:58.521 INFO:teuthology.orchestra.run.smithi045.stdout: "compression_type": "random" 2021-09-17T02:19:58.522 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:58.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:58.525 INFO:teuthology.orchestra.run.smithi045.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-17T02:19:58.525 INFO:teuthology.orchestra.run.smithi045.stdout: "index_type": 0 2021-09-17T02:19:58.525 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:58.525 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:58.526 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T02:19:58.526 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T02:19:58.526 INFO:teuthology.orchestra.run.smithi045.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-17T02:19:58.526 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:58.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:19:58.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:19:58.529 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T02:19:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:19:58.573 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:58.626 INFO:teuthology.orchestra.run.smithi045.stderr:Sending period to new master zone 95b0d488-6c59-4a37-b3ed-d5fa65f27678 2021-09-17T02:19:59.782 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:19:59.782 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:59.782 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 5, 2021-09-17T02:19:59.783 INFO:teuthology.orchestra.run.smithi045.stdout: "predecessor_uuid": "d9cd66ce-d16c-4226-9f7c-60489089c5d1", 2021-09-17T02:19:59.783 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": [], 2021-09-17T02:19:59.783 INFO:teuthology.orchestra.run.smithi045.stdout: "period_map": { 2021-09-17T02:19:59.783 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:19:59.784 INFO:teuthology.orchestra.run.smithi045.stdout: "zonegroups": [ 2021-09-17T02:19:59.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:59.784 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "test-zonegroup", 2021-09-17T02:19:59.784 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zonegroup", 2021-09-17T02:19:59.784 INFO:teuthology.orchestra.run.smithi045.stdout: "api_name": "", 2021-09-17T02:19:59.785 INFO:teuthology.orchestra.run.smithi045.stdout: "is_master": "true", 2021-09-17T02:19:59.785 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T02:19:59.785 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T02:19:59.785 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T02:19:59.785 INFO:teuthology.orchestra.run.smithi045.stdout: "hostnames": [], 2021-09-17T02:19:59.786 INFO:teuthology.orchestra.run.smithi045.stdout: "hostnames_s3website": [], 2021-09-17T02:19:59.786 INFO:teuthology.orchestra.run.smithi045.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:19:59.786 INFO:teuthology.orchestra.run.smithi045.stdout: "zones": [ 2021-09-17T02:19:59.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:59.787 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T02:19:59.787 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone3", 2021-09-17T02:19:59.787 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T02:19:59.787 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi013.front.sepia.ceph.com:8001" 2021-09-17T02:19:59.788 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T02:19:59.788 INFO:teuthology.orchestra.run.smithi045.stdout: "log_meta": "false", 2021-09-17T02:19:59.788 INFO:teuthology.orchestra.run.smithi045.stdout: "log_data": "true", 2021-09-17T02:19:59.788 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:19:59.788 INFO:teuthology.orchestra.run.smithi045.stdout: "read_only": "false", 2021-09-17T02:19:59.789 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_type": "", 2021-09-17T02:19:59.789 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from_all": "true", 2021-09-17T02:19:59.789 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from": [], 2021-09-17T02:19:59.789 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_zone": "" 2021-09-17T02:19:59.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:59.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:59.790 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T02:19:59.790 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone2", 2021-09-17T02:19:59.790 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T02:19:59.791 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi045.front.sepia.ceph.com:8000" 2021-09-17T02:19:59.791 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T02:19:59.791 INFO:teuthology.orchestra.run.smithi045.stdout: "log_meta": "false", 2021-09-17T02:19:59.791 INFO:teuthology.orchestra.run.smithi045.stdout: "log_data": "true", 2021-09-17T02:19:59.792 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:19:59.792 INFO:teuthology.orchestra.run.smithi045.stdout: "read_only": "false", 2021-09-17T02:19:59.792 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_type": "", 2021-09-17T02:19:59.792 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from_all": "false", 2021-09-17T02:19:59.792 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from": [], 2021-09-17T02:19:59.793 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_zone": "" 2021-09-17T02:19:59.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:59.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:59.793 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:19:59.793 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone1", 2021-09-17T02:19:59.794 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T02:19:59.794 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T02:19:59.794 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T02:19:59.794 INFO:teuthology.orchestra.run.smithi045.stdout: "log_meta": "false", 2021-09-17T02:19:59.795 INFO:teuthology.orchestra.run.smithi045.stdout: "log_data": "true", 2021-09-17T02:19:59.795 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:19:59.795 INFO:teuthology.orchestra.run.smithi045.stdout: "read_only": "false", 2021-09-17T02:19:59.795 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_type": "", 2021-09-17T02:19:59.795 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from_all": "true", 2021-09-17T02:19:59.796 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from": [], 2021-09-17T02:19:59.796 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_zone": "" 2021-09-17T02:19:59.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:59.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:59.796 INFO:teuthology.orchestra.run.smithi045.stdout: "id": "e8223d60-810a-44ad-a8ee-67d9caa7bcb0", 2021-09-17T02:19:59.797 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "test-zone4", 2021-09-17T02:19:59.797 INFO:teuthology.orchestra.run.smithi045.stdout: "endpoints": [ 2021-09-17T02:19:59.797 INFO:teuthology.orchestra.run.smithi045.stdout: "http://smithi045.front.sepia.ceph.com:8001" 2021-09-17T02:19:59.797 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T02:19:59.798 INFO:teuthology.orchestra.run.smithi045.stdout: "log_meta": "false", 2021-09-17T02:19:59.798 INFO:teuthology.orchestra.run.smithi045.stdout: "log_data": "true", 2021-09-17T02:19:59.798 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:19:59.798 INFO:teuthology.orchestra.run.smithi045.stdout: "read_only": "false", 2021-09-17T02:19:59.798 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_type": "pubsub", 2021-09-17T02:19:59.799 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from_all": "false", 2021-09-17T02:19:59.799 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_from": [ 2021-09-17T02:19:59.799 INFO:teuthology.orchestra.run.smithi045.stdout: "test-zone1" 2021-09-17T02:19:59.799 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T02:19:59.800 INFO:teuthology.orchestra.run.smithi045.stdout: "redirect_zone": "" 2021-09-17T02:19:59.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:59.800 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T02:19:59.800 INFO:teuthology.orchestra.run.smithi045.stdout: "placement_targets": [ 2021-09-17T02:19:59.800 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:59.801 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "default-placement", 2021-09-17T02:19:59.801 INFO:teuthology.orchestra.run.smithi045.stdout: "tags": [], 2021-09-17T02:19:59.801 INFO:teuthology.orchestra.run.smithi045.stdout: "storage_classes": [ 2021-09-17T02:19:59.801 INFO:teuthology.orchestra.run.smithi045.stdout: "STANDARD" 2021-09-17T02:19:59.801 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:59.802 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:59.802 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T02:19:59.802 INFO:teuthology.orchestra.run.smithi045.stdout: "default_placement": "default-placement", 2021-09-17T02:19:59.802 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T02:19:59.803 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_policy": { 2021-09-17T02:19:59.803 INFO:teuthology.orchestra.run.smithi045.stdout: "groups": [] 2021-09-17T02:19:59.803 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:59.804 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:59.804 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2021-09-17T02:19:59.804 INFO:teuthology.orchestra.run.smithi045.stdout: "short_zone_ids": [ 2021-09-17T02:19:59.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:59.816 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T02:19:59.817 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 4005434042 2021-09-17T02:19:59.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:59.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:59.817 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T02:19:59.818 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 4091795418 2021-09-17T02:19:59.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:59.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:59.818 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:19:59.819 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 803492335 2021-09-17T02:19:59.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:59.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:19:59.820 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "e8223d60-810a-44ad-a8ee-67d9caa7bcb0", 2021-09-17T02:19:59.820 INFO:teuthology.orchestra.run.smithi045.stdout: "val": 3317638424 2021-09-17T02:19:59.820 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:59.820 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:19:59.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:59.821 INFO:teuthology.orchestra.run.smithi045.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T02:19:59.821 INFO:teuthology.orchestra.run.smithi045.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:19:59.821 INFO:teuthology.orchestra.run.smithi045.stdout: "period_config": { 2021-09-17T02:19:59.821 INFO:teuthology.orchestra.run.smithi045.stdout: "bucket_quota": { 2021-09-17T02:19:59.822 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": false, 2021-09-17T02:19:59.822 INFO:teuthology.orchestra.run.smithi045.stdout: "check_on_raw": false, 2021-09-17T02:19:59.822 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size": -1, 2021-09-17T02:19:59.822 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size_kb": 0, 2021-09-17T02:19:59.823 INFO:teuthology.orchestra.run.smithi045.stdout: "max_objects": -1 2021-09-17T02:19:59.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:59.823 INFO:teuthology.orchestra.run.smithi045.stdout: "user_quota": { 2021-09-17T02:19:59.823 INFO:teuthology.orchestra.run.smithi045.stdout: "enabled": false, 2021-09-17T02:19:59.823 INFO:teuthology.orchestra.run.smithi045.stdout: "check_on_raw": false, 2021-09-17T02:19:59.824 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size": -1, 2021-09-17T02:19:59.824 INFO:teuthology.orchestra.run.smithi045.stdout: "max_size_kb": 0, 2021-09-17T02:19:59.824 INFO:teuthology.orchestra.run.smithi045.stdout: "max_objects": -1 2021-09-17T02:19:59.824 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:19:59.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:19:59.825 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T02:19:59.825 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_name": "test-realm", 2021-09-17T02:19:59.825 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:19:59.825 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:19:59.827 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-09-17T02:20:59.828 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-45 2021-09-17T02:20:59.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:20:59.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:20:59.901 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:20:59.950 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:20:59.951 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:20:59.991 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:21:00.003 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:21:00.004 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:21:00.004 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:00.005 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:21:00.005 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:21:00.005 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:21:00.005 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631845259.851939_401.1", 2021-09-17T02:21:00.005 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:20:59.851939Z" 2021-09-17T02:21:00.006 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:21:00.006 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:21:00.006 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:21:00.006 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:21:00.007 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:21:00.007 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:21:00.007 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:21:00.007 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:21:00.007 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:21:00.008 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:21:00.009 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:21:00.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:00.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:00.011 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:00.063 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:00.064 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:00.122 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:00.122 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:00.123 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:00.123 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:00.123 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:00.123 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:00.124 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:00.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:00.124 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:00.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:00.124 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:00.125 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:00.125 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:00.125 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:00.125 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:00.126 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:00.126 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:00.126 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:00.126 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:00.127 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:00.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:00.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:00.127 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:00.127 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:00.128 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:00.128 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:00.128 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:00.129 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:00.129 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:00.130 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:00.130 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:00.130 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:00.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:00.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:00.131 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:00.131 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:00.131 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:00.131 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:00.132 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:00.133 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:00.133 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:00.133 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:00.134 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:00.134 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:00.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:00.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:00.135 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:00.136 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:00.136 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:00.137 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:00.137 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:00.137 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:00.137 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:00.138 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:00.138 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:00.138 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:00.138 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:00.138 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:00.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:00.139 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:00.139 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:00.139 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:00.140 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:00.140 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:00.141 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:21:05.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:05.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:05.144 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:05.194 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:05.195 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:05.254 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:05.255 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:05.255 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:05.255 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:05.255 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:05.256 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:05.256 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:05.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:05.256 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:05.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:05.257 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:05.257 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:05.257 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:05.258 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:05.258 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:05.258 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:05.258 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:05.259 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:05.259 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:05.259 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:05.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:05.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:05.260 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:05.260 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:05.260 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:05.261 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:05.261 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:05.261 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:05.261 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:05.262 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:05.262 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:05.262 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:05.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:05.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:05.263 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:05.263 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:05.263 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:05.264 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:05.264 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:05.264 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:05.265 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:05.265 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:05.265 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:05.265 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:05.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:05.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:05.266 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:05.266 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:05.266 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:05.267 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:05.267 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:05.267 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:05.267 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:05.268 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:05.268 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:05.268 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:05.268 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:05.269 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:05.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:05.269 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:05.269 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:05.269 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:05.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:05.270 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:05.271 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:21:10.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:10.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:10.274 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:10.326 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:10.327 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:10.385 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:10.386 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:10.386 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:10.386 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:10.387 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:10.387 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:10.387 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:10.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:10.388 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:10.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:10.388 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:10.388 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:10.388 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:10.389 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:10.389 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:10.389 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:10.389 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:10.390 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:10.390 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:10.390 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:10.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:10.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:10.391 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:10.391 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:10.391 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:10.391 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:10.392 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:10.392 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:10.392 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:10.392 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:10.392 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:10.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:10.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:10.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:10.393 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:10.394 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:10.394 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:10.394 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:10.394 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:10.394 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:10.395 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:10.395 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:10.395 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:10.395 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:10.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:10.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:10.396 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:10.396 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:10.396 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:10.397 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:10.397 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:10.397 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:10.397 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:10.398 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:10.398 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:10.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:10.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:10.398 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:10.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:10.399 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:10.399 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:10.399 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:10.400 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:10.400 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:10.401 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:21:15.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:15.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:15.404 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:15.516 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:15.517 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:15.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:15.519 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:15.520 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:15.521 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:15.522 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:15.523 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:15.524 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:15.525 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:15.526 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:15.526 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:15.526 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:15.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:15.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:15.527 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:15.527 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:15.527 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:15.528 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:15.528 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:15.528 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:15.528 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:15.528 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:15.529 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:15.529 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:15.529 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:15.529 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:15.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:15.530 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:15.530 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:15.530 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:15.530 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:15.531 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:15.532 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:21:20.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:20.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:20.535 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:20.587 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:20.587 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:20.650 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:20.651 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:20.651 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:20.651 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:20.651 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:20.652 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:20.652 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:20.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:20.652 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:20.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:20.653 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:20.653 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:20.653 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:20.653 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:20.654 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:20.654 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:20.654 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:20.654 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:20.654 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:20.655 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:20.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:20.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:20.655 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:20.656 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:20.656 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:20.656 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:20.656 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:20.656 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:20.657 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:20.657 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:20.657 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:20.657 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:20.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:20.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:20.658 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:20.658 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:20.658 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:20.659 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:20.659 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:20.659 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:20.659 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:20.659 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:20.660 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:20.660 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:20.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:20.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:20.661 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:20.661 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:20.661 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:20.661 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:20.661 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:20.662 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:20.662 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:20.662 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:20.662 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:20.662 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:20.663 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:20.663 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:20.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:20.663 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:20.664 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:20.664 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:20.664 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:20.664 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:20.666 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:21:25.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:25.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:25.669 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:25.718 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:25.718 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:25.777 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:25.777 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:25.778 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:25.778 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:25.778 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:25.778 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:25.779 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:25.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:25.779 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:25.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:25.779 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:25.780 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:25.780 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:25.780 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:25.780 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:25.780 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:25.781 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:25.781 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:25.781 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:25.781 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:25.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:25.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:25.782 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:25.782 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:25.782 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:25.783 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:25.783 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:25.783 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:25.784 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:25.784 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:25.785 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:25.785 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:25.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:25.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:25.786 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:25.786 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:25.786 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:25.786 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:25.787 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:25.787 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:25.787 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:25.787 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:25.787 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:25.788 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:25.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:25.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:25.789 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:25.789 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:25.789 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:25.789 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:25.790 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:25.790 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:25.790 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:25.790 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:25.791 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:25.791 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:25.791 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:25.791 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:25.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:25.792 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:25.792 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:25.792 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:25.792 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:25.792 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:25.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:21:30.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:30.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:30.796 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:30.847 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:30.848 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:30.908 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:30.908 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:30.908 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:30.909 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:30.909 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:30.909 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:30.909 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:30.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:30.910 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:30.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:30.911 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:30.911 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:30.911 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:30.912 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:30.912 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:30.912 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:30.912 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:30.913 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:30.913 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:30.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:30.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:30.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:30.914 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:30.914 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:30.914 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:30.914 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:30.915 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:30.915 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:30.915 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:30.916 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:30.916 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:30.916 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:30.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:30.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:30.917 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:30.917 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:30.917 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:30.918 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:30.918 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:30.918 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:30.918 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:30.919 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:30.919 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:30.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:30.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:30.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:30.920 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:30.920 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:30.920 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:30.921 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:30.921 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:30.921 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:30.922 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:30.922 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:30.922 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:30.923 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:30.923 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:30.923 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:30.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:30.924 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:30.924 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:30.924 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:30.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:30.924 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:30.926 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:21:30.933 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:21:30.932+0000 7f1e1e02f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:21:30.936 INFO:tasks.rgw.c2.client.1.smithi045.stdout:2021-09-17T02:21:30.935+0000 7f4380417700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T02:21:35.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:35.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:35.929 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:35.979 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:35.980 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:36.040 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:36.041 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:36.042 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:36.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:36.044 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:36.045 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:36.046 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:36.047 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:36.048 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:36.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:36.050 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:36.051 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:36.052 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:36.053 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:36.054 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:36.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:36.056 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:36.056 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:36.058 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:21:41.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:41.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:41.062 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:41.112 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:41.112 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:41.172 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:41.173 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:41.173 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:41.173 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:41.173 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:41.174 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:41.174 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:41.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:41.174 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:41.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:41.175 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:41.175 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:41.175 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:41.176 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:41.176 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:41.176 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:41.176 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:41.176 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:41.177 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:41.177 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:41.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:41.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:41.178 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:41.178 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:41.178 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:41.178 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:41.179 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:41.179 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:41.179 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:41.179 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:41.179 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:41.180 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:41.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:41.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:41.180 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:41.180 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:41.181 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:41.181 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:41.181 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:41.181 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:41.182 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:41.182 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:41.182 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:41.182 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:41.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:41.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:41.183 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:41.183 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:41.183 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:41.184 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:41.184 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:41.184 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:41.184 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:41.184 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:41.185 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:41.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:41.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:41.185 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:41.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:41.186 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:41.186 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:41.186 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:41.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:41.187 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:41.188 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:21:46.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:46.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:46.190 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:46.241 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:46.241 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:46.302 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:46.302 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:46.303 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:46.303 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:46.303 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:46.303 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:46.304 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:46.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:46.304 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:46.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:46.304 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:46.305 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:46.305 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:46.305 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:46.305 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:46.306 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:46.306 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:46.306 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:46.307 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:46.307 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:46.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:46.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:46.307 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:46.308 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:46.308 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:46.308 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:46.308 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:46.309 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:46.309 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:46.309 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:46.309 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:46.309 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:46.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:46.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:46.310 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:46.310 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:46.311 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:46.311 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:46.311 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:46.311 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:46.311 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:46.312 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:46.312 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:46.312 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:46.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:46.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:46.313 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:46.313 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:46.313 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:46.314 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:46.314 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:46.314 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:46.314 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:46.315 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:46.315 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:46.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:46.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:46.315 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:46.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:46.316 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:46.316 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:46.316 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:46.317 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:46.317 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:46.318 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:21:51.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:51.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:51.321 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:51.371 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:51.371 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:51.432 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:51.432 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:51.433 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:51.433 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:51.433 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:51.433 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:51.434 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:51.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:51.434 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:51.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:51.434 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:51.435 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:51.435 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:51.435 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:51.435 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:51.436 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:51.436 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:51.436 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:51.436 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:51.436 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:51.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:51.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:51.437 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:51.437 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:51.438 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:51.438 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:51.438 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:51.438 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:51.438 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:51.439 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:51.439 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:51.439 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:51.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:51.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:51.440 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:51.440 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:51.440 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:51.440 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:51.441 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:51.441 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:51.441 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:51.441 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:51.441 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:51.442 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:51.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:51.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:51.442 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:51.443 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:51.443 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:51.443 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:51.443 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:51.443 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:51.444 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:51.444 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:51.444 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:51.444 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:51.444 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:51.445 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:51.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:51.445 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:51.445 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:51.446 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:51.446 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:51.446 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:51.447 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:21:56.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:56.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:21:56.450 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:21:56.501 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:21:56.501 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:21:56.563 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:21:56.563 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:21:56.564 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:21:56.564 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:21:56.564 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:21:56.564 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:21:56.565 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:56.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:56.565 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:21:56.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:56.566 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:21:56.566 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:56.566 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:56.566 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:21:56.566 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:56.567 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:21:56.567 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:56.567 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:21:56.567 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:56.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:56.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:56.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:56.568 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:21:56.568 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:56.569 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:56.569 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.064665_238.1", 2021-09-17T02:21:56.569 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:56.569 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:56.569 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:56.570 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.064665Z", 2021-09-17T02:21:56.570 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:56.570 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:56.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:56.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:56.571 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:21:56.571 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:56.571 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:56.572 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:56.572 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:56.572 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:56.573 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:56.573 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:56.573 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:56.573 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:56.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:56.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:21:56.574 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:21:56.574 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:21:56.574 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:21:56.575 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:21:56.575 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:21:56.575 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:21:56.575 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:21:56.575 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:21:56.576 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:21:56.576 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:56.576 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:56.576 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:21:56.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:21:56.577 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:21:56.577 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:21:56.577 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:21:56.578 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:21:56.578 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:21:56.579 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.064665_238.1 2021-09-17T02:22:01.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:01.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:01.581 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:01.625 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:01.626 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:01.681 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:01.681 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:01.681 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:01.682 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:01.682 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:01.683 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:01.683 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:01.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:01.683 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:01.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:01.684 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:01.684 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:01.684 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:01.685 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:01.685 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:01.685 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:01.685 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:01.685 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:01.686 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:01.686 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:01.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:01.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:01.687 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:01.687 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:01.687 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:01.687 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.288419_257.1", 2021-09-17T02:22:01.688 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:01.688 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:01.688 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:01.688 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.288419Z", 2021-09-17T02:22:01.689 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:01.689 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:01.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:01.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:01.690 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:01.690 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:01.690 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:01.690 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:01.690 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:01.691 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:01.691 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:01.691 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:01.691 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:01.692 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:01.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:01.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:01.692 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:01.693 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:01.693 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:01.693 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:01.693 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:01.693 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:01.694 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:01.694 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:01.694 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:01.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:01.695 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:01.695 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:01.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:01.695 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:01.695 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:01.696 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:01.696 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:01.696 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:01.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.288419_257.1 2021-09-17T02:22:06.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:06.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:06.700 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:06.744 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:06.800 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:06.801 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:06.801 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:06.801 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:06.801 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:06.802 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:06.802 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:06.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:06.802 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:06.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:06.803 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:06.803 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:06.803 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:06.804 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:06.804 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:06.804 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:06.804 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:06.804 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:06.805 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:06.805 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:06.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:06.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:06.806 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:06.806 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:06.806 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:06.807 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:06.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:06.809 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:06.810 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:06.811 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:06.812 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:06.812 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:06.812 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:06.812 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:06.812 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:06.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:06.814 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:06.816 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:22:11.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:11.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:11.818 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:11.863 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:11.863 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:11.917 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:11.917 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:11.917 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:11.918 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:11.918 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:11.918 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:11.918 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:11.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:11.919 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:11.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:11.919 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:11.919 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:11.920 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:11.920 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:11.920 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:11.920 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:11.921 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:11.921 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:11.921 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:11.921 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:11.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:11.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:11.922 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:11.922 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:11.922 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:11.923 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:22:11.923 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:11.923 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:11.923 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:11.923 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:22:11.924 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:11.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:11.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:11.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:11.924 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:11.925 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:11.925 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:11.925 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:11.925 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:11.926 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:11.926 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:11.926 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:11.926 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:11.926 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:11.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:11.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:11.927 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:11.927 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:11.928 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:11.928 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:11.928 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:11.928 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:11.929 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:11.929 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:11.929 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:11.929 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:11.930 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:11.930 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:11.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:11.930 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:11.930 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:11.931 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:11.931 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:11.931 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:11.932 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:22:16.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:16.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:16.935 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:16.983 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:16.984 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:17.041 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:17.041 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:17.042 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:17.042 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:17.043 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:17.043 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:17.043 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:17.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:17.044 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:17.044 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:17.044 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:17.044 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:17.045 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:17.045 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:17.045 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:17.045 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:17.046 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:17.046 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:17.046 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:17.046 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:17.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:17.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:17.047 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:17.047 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:17.047 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:17.047 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:22:17.048 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:17.048 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:17.048 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:17.048 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:22:17.049 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:17.049 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:17.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:17.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:17.049 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:17.050 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:17.050 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:17.050 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:17.050 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:17.051 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:17.051 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:17.051 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:17.051 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:17.051 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:17.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:17.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:17.052 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:17.052 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:17.053 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:17.053 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:17.053 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:17.053 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:17.053 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:17.054 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:17.054 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:17.054 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:17.054 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:17.055 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:17.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:17.055 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:17.055 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:17.055 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:17.056 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:17.056 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:17.057 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.088776_242.1 2021-09-17T02:22:22.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:22.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:22.060 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:22.110 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:22.110 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:22.167 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:22.168 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:22.168 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:22.168 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:22.169 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:22.169 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:22.169 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:22.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:22.170 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:22.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:22.170 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:22.170 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:22.171 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:22.171 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:22.171 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:22.171 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:22.172 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:22.172 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:22.172 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:22.172 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:22.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:22.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:22.173 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:22.173 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:22.173 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:22.174 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:22:22.174 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:22.174 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:22.174 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:22.175 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:22:22.175 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:22.175 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:22.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:22.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:22.176 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:22.176 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:22.176 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:22.177 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:22.177 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:22.177 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:22.177 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:22.177 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:22.178 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:22.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:22.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:22.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:22.179 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:22.179 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:22.179 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:22.179 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:22.180 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:22.180 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:22.180 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:22.180 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:22.180 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:22.181 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:22.181 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:22.181 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:22.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:22.182 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:22.182 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:22.182 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:22.182 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:22.183 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:22.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:22:27.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:27.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:27.187 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:27.237 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:27.237 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:27.296 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:27.296 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:27.296 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:27.297 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:27.297 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:27.297 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:27.297 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:27.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:27.298 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:27.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:27.298 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:27.299 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:27.299 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:27.299 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:27.299 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:27.299 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:27.300 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:27.300 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:27.300 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:27.300 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:27.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:27.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:27.301 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:27.301 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:27.301 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:27.302 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.091993_244.1", 2021-09-17T02:22:27.302 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:27.302 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:27.302 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:27.303 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.091993Z", 2021-09-17T02:22:27.303 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:27.303 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:27.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:27.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:27.304 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:27.304 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:27.304 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:27.305 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:27.305 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:27.305 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:27.306 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:27.306 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:27.306 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:27.306 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:27.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:27.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:27.307 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:27.307 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:27.307 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:27.308 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:27.308 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:27.308 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:27.308 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:27.308 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:27.309 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:27.309 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:27.309 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:27.309 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:27.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:27.310 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:27.310 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:27.310 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:27.310 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:27.311 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:27.312 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.091993_244.1 2021-09-17T02:22:32.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:32.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:32.314 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:32.364 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:32.365 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:32.424 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:32.424 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:32.425 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:32.425 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:32.425 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:32.425 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:32.426 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:32.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:32.426 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:32.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:32.427 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:32.427 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:32.427 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:32.427 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:32.428 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:32.428 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:32.428 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:32.428 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:32.428 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:32.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:32.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:32.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:32.429 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:32.430 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:32.430 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:32.430 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:22:32.430 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:32.430 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:32.431 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:32.431 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:22:32.431 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:32.431 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:32.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:32.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:32.432 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:32.432 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:32.432 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:32.433 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:32.433 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:32.433 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:32.433 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:32.433 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:32.434 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:32.434 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:32.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:32.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:32.435 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:32.435 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:32.435 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:32.435 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:32.435 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:32.436 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:32.436 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:32.436 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:32.436 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:32.436 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:32.437 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:32.437 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:32.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:32.437 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:32.438 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:32.438 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:32.438 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:32.438 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:32.439 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:22:37.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:37.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:37.442 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:37.555 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:37.555 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:37.556 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:37.556 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:37.556 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:37.556 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:37.557 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:37.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:37.557 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:37.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:37.558 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:37.558 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:37.558 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:37.558 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:37.559 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:37.559 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:37.559 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:37.559 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:37.559 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:37.560 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:37.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:37.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:37.560 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:37.561 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:37.561 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:37.561 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.272193_254.1", 2021-09-17T02:22:37.561 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:37.561 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:37.562 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:37.562 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.272193Z", 2021-09-17T02:22:37.562 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:37.562 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:37.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:37.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:37.563 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:37.563 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:37.564 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:37.564 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:37.564 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:37.564 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:37.565 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:37.565 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:37.565 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:37.565 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:37.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:37.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:37.566 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:37.566 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:37.566 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:37.567 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:37.567 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:37.567 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:37.567 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:37.570 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:37.571 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.272193_254.1 2021-09-17T02:22:42.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:42.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:42.574 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:42.626 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:42.626 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:42.688 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:42.688 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:42.688 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:42.689 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:42.689 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:42.689 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:42.689 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:42.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:42.690 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:42.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:42.690 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:42.691 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:42.691 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:42.691 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:42.691 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:42.691 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:42.692 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:42.692 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:42.692 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:42.692 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:42.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:42.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:42.693 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:42.693 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:42.693 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:42.694 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:22:42.694 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:42.694 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:42.694 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:42.694 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:22:42.695 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:42.695 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:42.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:42.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:42.696 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:42.696 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:42.696 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:42.696 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:42.696 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:42.697 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:42.697 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:42.697 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:42.697 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:42.697 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:42.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:42.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:42.698 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:42.698 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:42.699 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:42.699 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:42.699 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:42.699 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:42.700 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:42.700 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:42.700 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:42.700 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:42.701 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:42.701 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:42.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:42.701 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:42.702 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:42.702 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:42.702 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:42.702 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:42.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.086132_241.1 2021-09-17T02:22:47.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:47.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:47.707 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:47.756 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:47.757 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:47.817 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:47.818 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:47.818 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:47.818 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:47.818 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:47.819 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:47.819 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:47.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:47.819 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:47.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:47.820 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:47.820 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:47.820 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:47.820 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:47.821 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:47.821 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:47.821 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:47.821 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:47.821 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:47.822 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:47.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:47.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:47.822 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:47.823 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:47.823 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:47.823 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:22:47.823 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:47.823 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:47.824 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:47.824 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:22:47.824 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:47.824 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:47.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:47.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:47.825 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:47.825 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:47.825 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:47.826 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:47.826 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:47.826 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:47.826 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:47.826 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:47.827 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:47.827 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:47.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:47.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:47.828 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:47.828 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:47.828 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:47.828 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:47.828 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:47.829 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:47.829 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:47.829 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:47.829 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:47.829 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:47.830 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:47.830 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:47.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:47.830 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:47.831 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:47.831 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:47.831 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:47.831 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:47.832 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.088776_242.1 2021-09-17T02:22:52.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:52.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:52.835 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:52.884 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:52.884 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:52.946 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:52.946 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:52.947 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:52.948 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:52.949 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:52.949 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:52.949 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:52.949 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:52.949 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:52.950 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:52.950 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:52.950 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:52.950 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:52.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:52.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:52.951 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:52.951 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:52.951 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:52.952 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:22:52.952 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:52.952 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:52.952 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:52.953 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:22:52.953 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:52.953 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:52.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:52.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:52.954 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:52.954 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:52.954 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:52.954 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:52.955 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:52.955 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:52.955 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:52.955 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:52.956 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:52.956 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:52.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:52.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:52.957 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:52.957 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:52.957 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:52.957 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:52.958 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:52.958 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:52.958 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:52.958 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:52.959 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:52.959 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:52.959 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:52.959 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:52.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:52.960 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:52.960 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:52.960 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:52.960 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:52.961 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:52.961 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:22:57.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:57.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:22:57.964 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:22:58.032 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:22:58.032 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:22:58.426 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:22:58.426 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:22:58.426 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:22:58.427 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:22:58.427 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:22:58.427 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:22:58.427 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:58.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:58.428 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:22:58.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:58.428 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:22:58.428 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:58.429 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:58.429 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:22:58.429 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:58.429 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:22:58.430 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:58.430 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:22:58.430 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:58.430 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:58.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:58.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:58.431 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:22:58.431 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:58.431 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:58.432 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:22:58.432 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:58.432 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:58.432 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:58.433 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:22:58.433 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:58.433 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:58.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:58.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:58.434 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:22:58.434 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:58.434 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:58.434 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:58.435 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:58.435 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:58.435 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:58.435 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:58.436 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:58.436 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:58.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:58.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:22:58.436 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:22:58.437 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:22:58.437 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:22:58.437 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:22:58.437 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:22:58.438 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:22:58.438 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:22:58.438 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:22:58.438 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:22:58.439 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:58.439 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:58.439 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:22:58.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:22:58.439 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:22:58.440 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:22:58.440 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:22:58.440 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:22:58.440 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:22:58.441 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.086132_241.1 2021-09-17T02:23:03.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:03.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:03.444 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:23:03.525 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:23:03.526 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:23:03.636 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:23:03.637 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:23:03.637 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:23:03.637 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:23:03.637 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:23:03.638 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:23:03.638 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:03.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:03.638 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:23:03.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:03.639 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:23:03.639 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:03.639 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:03.639 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:23:03.640 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:03.640 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:23:03.640 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:03.640 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:23:03.641 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:03.641 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:03.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:03.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:03.641 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:23:03.642 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:03.642 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:03.642 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.105759_246.1", 2021-09-17T02:23:03.642 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:03.643 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:03.643 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:03.643 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.105759Z", 2021-09-17T02:23:03.643 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:03.643 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:03.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:03.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:03.644 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:23:03.644 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:03.645 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:03.645 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:03.645 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:03.645 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:03.645 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:03.646 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:03.646 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:03.646 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:03.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:03.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:03.647 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:23:03.647 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:03.647 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:03.648 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:03.648 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:03.648 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:03.648 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:03.648 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:03.649 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:03.649 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:03.649 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:03.649 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:23:03.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:03.650 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:23:03.650 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:23:03.650 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:23:03.650 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:03.651 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:23:03.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.105759_246.1 2021-09-17T02:23:08.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:08.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:08.654 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:23:08.698 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:23:08.699 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:23:08.764 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:23:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:23:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:23:08.765 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:23:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:23:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:23:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:08.766 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:23:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:23:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:08.767 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:23:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:23:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:08.768 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:23:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:08.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:23:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.091993_244.1", 2021-09-17T02:23:08.770 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:08.771 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.091993Z", 2021-09-17T02:23:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:08.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:23:08.773 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:08.774 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:08.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:23:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:08.776 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:08.777 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:08.777 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:08.777 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:08.777 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:08.777 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:08.778 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:08.778 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:08.778 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:23:08.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:08.779 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:23:08.779 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:23:08.779 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:23:08.779 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:08.779 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:23:08.781 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.091993_244.1 2021-09-17T02:23:13.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:13.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:13.783 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:23:13.834 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:23:13.835 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:23:14.057 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:23:14.057 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:23:14.057 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:23:14.057 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:23:14.058 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:23:14.058 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:23:14.058 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:14.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:14.059 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:23:14.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:14.060 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:23:14.060 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:14.060 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:14.060 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:23:14.061 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:14.061 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:23:14.061 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:14.061 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:23:14.062 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:14.062 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:14.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:14.062 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:14.063 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:23:14.063 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:14.063 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:14.063 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.103309_245.1", 2021-09-17T02:23:14.063 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:14.064 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:14.064 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:14.064 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.103309Z", 2021-09-17T02:23:14.064 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:14.064 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:14.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:14.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:14.065 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:23:14.065 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:14.066 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:14.066 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:14.066 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:14.066 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:14.066 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:14.067 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:14.067 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:14.067 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:14.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:14.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:14.068 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:23:14.068 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:14.068 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:14.068 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:14.069 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:14.069 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:14.070 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:14.070 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:14.070 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:14.071 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:14.071 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:14.071 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:23:14.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:14.071 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:23:14.072 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:23:14.072 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:23:14.072 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:14.072 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:23:14.073 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.103309_245.1 2021-09-17T02:23:19.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:19.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:19.075 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:23:19.125 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:23:19.125 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:23:19.178 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:23:19.179 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:23:19.179 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:23:19.179 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:23:19.179 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:23:19.180 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:23:19.180 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:19.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:19.180 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:23:19.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:19.181 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:23:19.181 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:19.181 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:19.181 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:23:19.182 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:19.182 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:23:19.182 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:19.182 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:23:19.183 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:19.183 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:19.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:19.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:19.183 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:23:19.184 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:19.184 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:19.184 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.272193_254.1", 2021-09-17T02:23:19.184 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:19.185 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:19.185 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:19.185 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.272193Z", 2021-09-17T02:23:19.185 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:19.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:19.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:19.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:19.186 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:23:19.186 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:19.186 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:19.187 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:19.187 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:19.187 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:19.187 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:19.188 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:19.188 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:19.188 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:19.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:19.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:19.189 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:23:19.189 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:19.189 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:19.189 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:19.190 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:19.190 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:19.190 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:19.190 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:19.190 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:19.191 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:19.191 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:19.191 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:23:19.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:19.191 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:23:19.192 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:23:19.192 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:23:19.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:19.192 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:23:19.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.272193_254.1 2021-09-17T02:23:24.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:24.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:24.196 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:23:24.243 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:23:24.243 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:23:24.298 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:23:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:23:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:23:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:23:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:23:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:23:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:23:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:23:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:23:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:23:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:23:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:23:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:23:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:23:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:23:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:23:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:23:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:23:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:23:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:23:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:24.312 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:23:24.313 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.089761_243.1 2021-09-17T02:23:29.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:29.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:29.316 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:23:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:23:29.360 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:23:29.414 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:23:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:23:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:23:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:23:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:23:29.415 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:23:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:23:29.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:23:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:29.417 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:23:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:23:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:23:29.418 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:29.419 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:23:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.161433_248.1", 2021-09-17T02:23:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:29.420 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.161433Z", 2021-09-17T02:23:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:29.421 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:29.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:29.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:29.422 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:23:29.422 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:29.422 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:29.423 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:29.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:23:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:29.425 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:29.426 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:29.427 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:23:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:23:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:23:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:23:29.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:23:29.430 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.161433_248.1 2021-09-17T02:23:34.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:34.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:34.432 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:23:34.476 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:23:34.476 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:23:34.531 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:23:34.531 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:23:34.531 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:23:34.532 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:23:34.532 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:23:34.532 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:23:34.532 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:34.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:34.533 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:23:34.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:34.533 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:23:34.533 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:34.534 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:34.534 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:23:34.534 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:34.534 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:23:34.535 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:34.535 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:23:34.535 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:34.535 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:34.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:34.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:34.536 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:23:34.536 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:34.536 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:34.537 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.288419_257.1", 2021-09-17T02:23:34.537 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:34.537 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:34.537 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:34.538 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.288419Z", 2021-09-17T02:23:34.538 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:34.538 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:34.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:34.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:34.539 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:23:34.539 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:34.540 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:34.540 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:34.540 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:34.540 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:34.540 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:34.541 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:34.541 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:34.541 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:34.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:34.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:34.542 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:23:34.542 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:34.542 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:34.542 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:34.543 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:34.543 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:34.543 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:34.543 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:34.544 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:34.544 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:34.544 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:34.544 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:23:34.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:34.545 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:23:34.545 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:23:34.545 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:23:34.545 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:34.545 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:23:34.547 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.288419_257.1 2021-09-17T02:23:39.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:39.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:39.549 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:23:39.593 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:23:39.594 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:23:39.649 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:23:39.649 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:23:39.650 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:23:39.650 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:23:39.650 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:23:39.650 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:23:39.651 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:39.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:39.651 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:23:39.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:39.652 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:23:39.652 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:39.652 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:39.652 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:23:39.652 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:39.653 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:23:39.653 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:39.653 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:23:39.653 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:23:39.654 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:23:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:39.655 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:23:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:39.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:23:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:39.657 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:39.658 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:39.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:23:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:39.660 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:39.661 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:23:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:39.662 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:23:39.663 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:23:39.663 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:23:39.663 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:39.663 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:23:39.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.086132_241.1 2021-09-17T02:23:44.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:44.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:44.667 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:23:44.711 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:23:44.711 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:23:44.764 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:23:44.765 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:23:44.765 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:23:44.765 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:23:44.766 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:23:44.766 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:23:44.766 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:44.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:44.766 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:23:44.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:44.767 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:23:44.767 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:44.767 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:44.767 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:23:44.768 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:44.768 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:23:44.768 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:44.768 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:23:44.768 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:44.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:44.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:44.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:44.769 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:23:44.770 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:44.770 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:44.770 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844261.852846_289.1", 2021-09-17T02:23:44.770 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:44.771 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:44.771 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:44.771 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:21.852846Z", 2021-09-17T02:23:44.772 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:44.772 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:44.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:44.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:44.772 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:23:44.773 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:44.773 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:44.773 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:44.773 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:44.774 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:44.774 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:44.774 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:44.774 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:44.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:44.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:44.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:44.775 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:23:44.775 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:44.775 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:44.776 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:44.776 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:44.776 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:44.776 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:44.776 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:44.777 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:44.777 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:44.777 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:44.777 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:23:44.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:44.778 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:23:44.778 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:23:44.778 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:23:44.778 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:44.779 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:23:44.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844261.852846_289.1 2021-09-17T02:23:49.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:49.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:49.782 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:23:49.828 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:23:49.828 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:23:49.882 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:23:49.882 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:23:49.882 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:23:49.883 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:23:49.883 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:23:49.883 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:23:49.883 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:49.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:49.884 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:23:49.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:49.884 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:23:49.884 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:49.885 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:49.885 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:23:49.885 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:49.885 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:23:49.886 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:49.886 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:23:49.886 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:49.886 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:49.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:49.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:49.887 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:23:49.887 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:49.887 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:49.888 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.091993_244.1", 2021-09-17T02:23:49.888 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:49.888 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:49.888 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:49.888 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.091993Z", 2021-09-17T02:23:49.889 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:49.889 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:49.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:49.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:49.889 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:23:49.890 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:49.890 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:49.890 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:49.890 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:49.890 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:49.891 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:49.891 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:49.891 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:49.891 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:49.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:49.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:49.892 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:23:49.892 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:49.892 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:49.893 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:49.893 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:49.893 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:49.893 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:49.893 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:49.894 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:49.894 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:49.894 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:49.894 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:23:49.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:49.895 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:23:49.895 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:23:49.895 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:23:49.895 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:49.896 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:23:49.897 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.091993_244.1 2021-09-17T02:23:54.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:54.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:23:54.900 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:23:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:23:54.944 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:23:54.997 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:23:54.997 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:23:54.997 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:23:54.997 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:23:54.998 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:23:54.998 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:23:54.998 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:54.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:54.999 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:23:54.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:54.999 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:23:54.999 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:55.000 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:55.000 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:23:55.000 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:55.000 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:23:55.001 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:55.001 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:23:55.001 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:55.001 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:55.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:55.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:55.002 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:23:55.002 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:55.002 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:55.003 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:23:55.003 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:55.003 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:55.003 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:55.004 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:23:55.004 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:55.004 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:55.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:55.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:55.005 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:23:55.005 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:55.005 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:55.005 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:55.006 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:55.006 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:55.006 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:55.006 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:55.006 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:55.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:55.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:55.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:23:55.007 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:23:55.008 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:23:55.008 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:23:55.008 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:23:55.008 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:23:55.008 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:23:55.009 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:23:55.009 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:23:55.009 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:23:55.009 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:55.009 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:55.010 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:23:55.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:23:55.010 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:23:55.010 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:23:55.010 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:23:55.011 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:23:55.011 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:23:55.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.067914_240.1 2021-09-17T02:24:00.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:00.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:00.015 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:00.060 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:00.061 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:00.113 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:00.114 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:00.114 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:00.114 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:00.115 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:00.115 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:00.115 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:00.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:00.115 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:00.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:00.116 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:00.116 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:00.116 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:00.117 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:00.117 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:00.117 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:00.117 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:00.117 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:00.118 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:00.118 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:00.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:00.118 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:00.119 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:00.119 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:00.119 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:00.119 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.288419_257.1", 2021-09-17T02:24:00.119 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:00.120 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:00.120 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:00.120 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.288419Z", 2021-09-17T02:24:00.120 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:00.120 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:00.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:00.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:00.121 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:00.121 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:00.122 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:00.122 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:00.122 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:00.122 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:00.123 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:00.123 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:00.123 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:00.123 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:00.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:00.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:00.124 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:00.124 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:00.124 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:00.124 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:00.125 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:00.125 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:00.125 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:00.125 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:00.125 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:00.126 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:00.126 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:00.126 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:00.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:00.127 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:00.127 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:00.127 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:00.127 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:00.128 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:00.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.288419_257.1 2021-09-17T02:24:05.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:05.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:05.132 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:05.180 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:05.181 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:05.239 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:05.239 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:05.239 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:05.239 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:05.243 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:05.243 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:05.243 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:05.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:05.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:05.244 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:05.244 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:05.244 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:05.244 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.268651_252.1", 2021-09-17T02:24:05.245 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:05.245 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:05.245 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:05.245 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.268651Z", 2021-09-17T02:24:05.246 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:05.246 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:05.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:05.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:05.247 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:05.247 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:05.247 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:05.247 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:05.247 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:05.248 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:05.248 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:05.248 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:05.248 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:05.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:05.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:05.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:05.249 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:05.249 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:05.250 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:05.250 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:05.250 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:05.250 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:05.250 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:05.251 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:05.251 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:05.251 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:05.251 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:05.252 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:05.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:05.252 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:05.252 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:05.253 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:05.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:05.253 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:05.254 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.268651_252.1 2021-09-17T02:24:10.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:10.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:10.257 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:10.302 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:10.303 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:10.357 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:10.358 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:10.358 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:10.358 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:10.358 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:10.359 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:10.359 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:10.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:10.359 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:10.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:10.360 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:10.360 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:10.360 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:10.360 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:10.360 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:10.361 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:10.361 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:10.361 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:10.361 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:10.362 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:10.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:10.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:10.362 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:10.362 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:10.363 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:10.363 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:24:10.363 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:10.363 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:10.363 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:10.364 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:24:10.364 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:10.364 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:10.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:10.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:10.365 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:10.365 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:10.365 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:10.365 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:10.366 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:10.366 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:10.366 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:10.366 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:10.367 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:10.367 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:10.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:10.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:10.367 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:10.368 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:10.368 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:10.368 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:10.368 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:10.368 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:10.369 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:10.369 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:10.369 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:10.369 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:10.370 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:10.370 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:10.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:10.370 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:10.371 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:10.371 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:10.371 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:10.371 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:10.372 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:24:15.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:15.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:15.376 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:15.424 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:15.424 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:15.481 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:15.481 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:15.481 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:15.482 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:15.482 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:15.482 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:15.482 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:15.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:15.483 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:15.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:15.483 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:15.484 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:15.484 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:15.484 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:15.484 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:15.484 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:15.485 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:15.485 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:15.485 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:15.485 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:15.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:15.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:15.486 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:15.486 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:15.486 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:15.487 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.268651_252.1", 2021-09-17T02:24:15.487 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:15.487 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:15.487 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:15.487 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.268651Z", 2021-09-17T02:24:15.488 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:15.488 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:15.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:15.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:15.488 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:15.489 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:15.489 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:15.489 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:15.489 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:15.489 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:15.490 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:15.490 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:15.490 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:15.490 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:15.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:15.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:15.491 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:15.491 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:15.491 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:15.492 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:15.492 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:15.492 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:15.492 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:15.492 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:15.493 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:15.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:15.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:15.493 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:15.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:15.494 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:15.494 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:15.494 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:15.494 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:15.495 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:15.496 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.268651_252.1 2021-09-17T02:24:20.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:20.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:20.500 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:20.545 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:20.602 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:20.602 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:20.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:20.604 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:20.605 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:20.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:20.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:20.607 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:20.607 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:20.607 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:20.607 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.212386_250.1", 2021-09-17T02:24:20.608 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:20.608 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:20.608 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:20.608 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.212386Z", 2021-09-17T02:24:20.609 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:20.609 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:20.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:20.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:20.610 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:20.610 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:20.610 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:20.610 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:20.610 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:20.611 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:20.611 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:20.611 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:20.611 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:20.612 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:20.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:20.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:20.612 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:20.612 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:20.613 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:20.613 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:20.613 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:20.613 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:20.613 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:20.614 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:20.614 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:20.614 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:20.614 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:20.615 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:20.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:20.615 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:20.615 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:20.615 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:20.616 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:20.616 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:20.617 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.212386_250.1 2021-09-17T02:24:25.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:25.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:25.620 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:25.666 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:25.666 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:25.720 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:25.720 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:25.720 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:25.721 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:25.721 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:25.721 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:25.721 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:25.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:25.722 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:25.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:25.722 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:25.722 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:25.723 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:25.723 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:25.723 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:25.723 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:25.724 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:25.724 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:25.724 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:25.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:25.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:25.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:25.725 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:25.725 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:25.725 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:25.726 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:24:25.726 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:25.726 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:25.726 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:25.726 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:24:25.727 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:25.727 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:25.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:25.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:25.727 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:25.728 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:25.728 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:25.728 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:25.728 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:25.729 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:25.729 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:25.729 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:25.729 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:25.730 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:25.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:25.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:25.730 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:25.730 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:25.731 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:25.731 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:25.731 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:25.731 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:25.732 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:25.732 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:25.732 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:25.732 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:25.732 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:25.733 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:25.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:25.733 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:25.733 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:25.733 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:25.734 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:25.734 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:25.735 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:24:30.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:30.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:30.739 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:30.785 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:30.786 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:30.839 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:30.840 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:30.840 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:30.841 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:30.841 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:30.842 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:30.842 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:30.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:30.844 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:30.844 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:30.845 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:30.845 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:30.845 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:30.846 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:30.846 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:30.847 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:30.847 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:30.848 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:30.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:30.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:30.849 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:30.849 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:30.850 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:30.851 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.157882_247.1", 2021-09-17T02:24:30.851 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:30.851 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:30.852 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:30.852 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.157882Z", 2021-09-17T02:24:30.852 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:30.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:30.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:30.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:30.853 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:30.853 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:30.853 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:30.854 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:30.854 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:30.854 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:30.854 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:30.854 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:30.855 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:30.855 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:30.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:30.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:30.856 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:30.856 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:30.856 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:30.856 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:30.856 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:30.857 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:30.857 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:30.857 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:30.857 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:30.857 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:30.858 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:30.858 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:30.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:30.858 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:30.859 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:30.859 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:30.859 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:30.859 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:30.860 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.157882_247.1 2021-09-17T02:24:35.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:35.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:35.863 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:35.914 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:35.914 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:35.975 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:35.975 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:35.976 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:35.977 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:35.978 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:35.979 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:35.980 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.105759_246.1", 2021-09-17T02:24:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:35.981 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.105759Z", 2021-09-17T02:24:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:35.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:35.983 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:35.984 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:35.985 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:35.986 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:35.987 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:35.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:35.989 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:35.990 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.105759_246.1 2021-09-17T02:24:40.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:40.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:40.993 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:41.043 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:41.043 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:41.103 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:41.103 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:41.103 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:41.104 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:41.104 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:41.104 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:41.104 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:41.105 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:41.105 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:41.105 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:41.105 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:41.105 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:41.106 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:41.106 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:41.106 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:41.106 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:41.107 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:41.107 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:41.107 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:41.107 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:41.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:41.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:41.108 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:41.108 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:41.109 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:41.109 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.209532_249.1", 2021-09-17T02:24:41.109 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:41.109 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:41.109 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:41.110 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.209532Z", 2021-09-17T02:24:41.110 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:41.110 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:41.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:41.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:41.111 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:41.111 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:41.111 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:41.112 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:41.112 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:41.112 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:41.112 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:41.112 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:41.113 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:41.113 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:41.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:41.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:41.114 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:41.114 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:41.114 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:41.114 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:41.114 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:41.115 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:41.115 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:41.116 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:41.116 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:41.116 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:41.116 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:41.117 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:41.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:41.117 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:41.117 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:41.118 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:41.118 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:41.118 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:41.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.209532_249.1 2021-09-17T02:24:46.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:46.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:46.122 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:46.173 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:46.235 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:46.235 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:46.235 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:46.236 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:46.236 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:46.236 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:46.236 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:46.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:46.237 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:46.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:46.237 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:46.237 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:46.238 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:46.238 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:46.238 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:46.238 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:46.239 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:46.239 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:46.239 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:46.239 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:46.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:46.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:46.240 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:46.240 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:46.241 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:46.241 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.266133_251.1", 2021-09-17T02:24:46.241 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:46.241 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:46.241 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:46.242 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.266133Z", 2021-09-17T02:24:46.242 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:46.242 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:46.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:46.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:46.243 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:46.243 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:46.243 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:46.244 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:46.244 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:46.244 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:46.244 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:46.245 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:46.245 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:46.245 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:46.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:46.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:46.246 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:46.246 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:46.246 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:46.246 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:46.247 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:46.247 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:46.247 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:46.247 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:46.247 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:46.248 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:46.248 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:46.248 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:46.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:46.248 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:46.249 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:46.249 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:46.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:46.249 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:46.251 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.266133_251.1 2021-09-17T02:24:51.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:51.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:51.253 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:51.304 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:51.305 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:51.368 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:51.368 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:51.368 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:51.368 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:51.369 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:51.369 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:51.369 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:51.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:51.370 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:51.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:51.370 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:51.370 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:51.370 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:51.371 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:51.371 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:51.371 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:51.371 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:51.372 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:51.372 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:51.372 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:51.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:51.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:51.373 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:51.373 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:51.373 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:51.373 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.209532_249.1", 2021-09-17T02:24:51.373 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:51.374 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:51.374 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:51.374 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.209532Z", 2021-09-17T02:24:51.374 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:51.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:51.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:51.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:51.375 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:51.375 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:51.376 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:51.376 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:51.376 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:51.376 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:51.376 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:51.377 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:51.377 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:51.377 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:51.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:51.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:51.378 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:51.378 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:51.378 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:51.378 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:51.379 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:51.379 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:51.379 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:51.379 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:51.380 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:51.380 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:51.380 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:51.380 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:51.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:51.381 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:51.381 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:51.381 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:51.381 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:51.381 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:51.382 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.209532_249.1 2021-09-17T02:24:56.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:56.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:24:56.386 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:24:56.437 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:24:56.437 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:24:56.513 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:24:56.514 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:24:56.514 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:24:56.514 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:24:56.515 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:24:56.515 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:24:56.515 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:56.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:56.515 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:24:56.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:56.516 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:24:56.516 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:56.516 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:56.517 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:24:56.517 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:56.517 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:24:56.517 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:56.517 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:24:56.518 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:56.518 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:56.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:56.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:56.519 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:24:56.519 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:56.519 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:56.519 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:24:56.519 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:56.520 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:56.520 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:56.520 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:24:56.520 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:56.520 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:56.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:56.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:56.521 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:24:56.521 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:56.521 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:56.522 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:56.522 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:56.522 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:56.522 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:56.522 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:56.523 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:56.523 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:56.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:56.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:24:56.524 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:24:56.524 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:24:56.524 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:24:56.524 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:24:56.524 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:24:56.525 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:24:56.525 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:24:56.525 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:24:56.525 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:24:56.525 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:56.526 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:56.526 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:24:56.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:24:56.526 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:24:56.527 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:24:56.527 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:24:56.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:24:56.527 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:24:56.528 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.086132_241.1 2021-09-17T02:25:01.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:01.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:01.531 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:01.602 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:01.602 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:02.013 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:02.013 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:02.014 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:02.014 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:02.014 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:02.014 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:02.015 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:02.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:02.015 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:02.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:02.016 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:02.016 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:02.016 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:02.016 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:02.017 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:02.017 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:02.017 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:02.017 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:02.017 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:02.018 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:02.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:02.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:02.018 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:02.019 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:02.019 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:02.019 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.103309_245.1", 2021-09-17T02:25:02.020 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:02.020 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:02.020 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:02.020 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.103309Z", 2021-09-17T02:25:02.021 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:02.021 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:02.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:02.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:02.021 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:02.022 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:02.022 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:02.022 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:02.022 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:02.023 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:02.023 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:02.023 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:02.023 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:02.024 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:02.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:02.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:02.024 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:02.024 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:02.025 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:02.025 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:02.025 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:02.025 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:02.026 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:02.026 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:02.026 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:02.026 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:02.026 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:02.027 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:02.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:02.027 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:02.027 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:02.028 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:02.028 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:02.028 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:02.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.103309_245.1 2021-09-17T02:25:07.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:07.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:07.033 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:07.115 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:07.115 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:07.198 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:07.198 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:07.198 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:07.199 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:07.199 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:07.199 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:07.199 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:07.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:07.200 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:07.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:07.200 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:07.201 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:07.201 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:07.201 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:07.201 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:07.202 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:07.202 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:07.202 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:07.202 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:07.203 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:07.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:07.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:07.203 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:07.204 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:07.204 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:07.204 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:25:07.205 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:07.205 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:07.205 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:07.205 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:25:07.206 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:07.206 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:07.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:07.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:07.207 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:07.207 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:07.208 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:07.208 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:07.208 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:07.208 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:07.209 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:07.209 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:07.209 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:07.209 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:07.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:07.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:07.210 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:07.210 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:07.211 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:07.211 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:07.211 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:07.211 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:07.211 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:07.212 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:07.212 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:07.212 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:07.212 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:07.213 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:07.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:07.213 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:07.213 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:07.214 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:07.214 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:07.214 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:07.215 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.089761_243.1 2021-09-17T02:25:12.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:12.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:12.218 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:12.312 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:12.313 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:12.455 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:12.455 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:12.455 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:12.456 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:12.456 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:12.456 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:12.456 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:12.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:12.457 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:12.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:12.457 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:12.457 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:12.458 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:12.458 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:12.458 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:12.458 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:12.458 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:12.459 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:12.459 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:12.459 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:12.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:12.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:12.460 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:12.460 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:12.460 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:12.460 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:25:12.461 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:12.461 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:12.461 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:12.461 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:25:12.461 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:12.462 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:12.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:12.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:12.462 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:12.462 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:12.463 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:12.463 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:12.463 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:12.463 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:12.463 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:12.464 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:12.464 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:12.464 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:12.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:12.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:12.465 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:12.465 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:12.465 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:12.465 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:12.465 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:12.466 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:12.466 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:12.466 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:12.466 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:12.467 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:12.467 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:12.467 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:12.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:12.468 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:12.468 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:12.468 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:12.469 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:12.469 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:12.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.067914_240.1 2021-09-17T02:25:17.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:17.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:17.474 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:17.522 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:17.522 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:17.697 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:17.698 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:17.698 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:17.698 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:17.698 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:17.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:17.700 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:17.701 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:17.702 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:17.703 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.212386_250.1", 2021-09-17T02:25:17.704 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.212386Z", 2021-09-17T02:25:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:17.705 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:17.706 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:17.707 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:17.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:17.709 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:17.710 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:17.710 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:17.710 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:17.710 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:17.711 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:17.711 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:17.711 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:17.711 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:17.711 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:17.712 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:17.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:17.712 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:17.712 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:17.713 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:17.714 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:17.714 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:17.715 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.212386_250.1 2021-09-17T02:25:22.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:22.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:22.718 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:22.791 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:22.792 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:23.057 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:23.058 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:23.058 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:23.058 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:23.059 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:23.059 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:23.059 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:23.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:23.059 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:23.060 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:23.060 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:23.060 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:23.060 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:23.061 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:23.061 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:23.061 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:23.061 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:23.062 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:23.062 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:23.062 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:23.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:23.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:23.063 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:23.063 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:23.063 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:23.064 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.272193_254.1", 2021-09-17T02:25:23.064 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:23.064 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:23.064 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:23.064 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.272193Z", 2021-09-17T02:25:23.065 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:23.065 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:23.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:23.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:23.066 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:23.066 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:23.066 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:23.066 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:23.067 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:23.067 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:23.067 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:23.067 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:23.068 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:23.068 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:23.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:23.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:23.068 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:23.069 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:23.069 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:23.069 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:23.069 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:23.070 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:23.070 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:23.070 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:23.070 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:23.071 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:23.071 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:23.071 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:23.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:23.072 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:23.072 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:23.072 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:23.072 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:23.073 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:23.074 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.272193_254.1 2021-09-17T02:25:28.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:28.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:28.077 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:28.122 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:28.123 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:28.204 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:28.204 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:28.204 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:28.205 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:28.205 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:28.205 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:28.205 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:28.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:28.206 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:28.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:28.206 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:28.207 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:28.207 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:28.207 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:28.207 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:28.208 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:28.208 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:28.208 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:28.208 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:28.208 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:28.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:28.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:28.209 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:28.209 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:28.210 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:28.210 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:25:28.210 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:28.210 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:28.210 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:28.211 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:25:28.211 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:28.211 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:28.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:28.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:28.212 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:28.212 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:28.213 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:28.213 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:28.213 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:28.213 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:28.213 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:28.214 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:28.214 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:28.214 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:28.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:28.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:28.215 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:28.215 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:28.215 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:28.215 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:28.216 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:28.216 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:28.216 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:28.216 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:28.217 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:28.217 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:28.217 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:28.217 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:28.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:28.218 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:28.218 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:28.218 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:28.219 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:28.219 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:28.220 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.086132_241.1 2021-09-17T02:25:33.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:33.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:33.222 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:33.267 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:33.267 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:33.321 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:33.321 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:33.321 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:33.322 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:33.322 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:33.322 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:33.322 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:33.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:33.323 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:33.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:33.323 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:33.323 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:33.324 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:33.324 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:33.324 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:33.324 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:33.324 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:33.325 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:33.325 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:33.325 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:33.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:33.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:33.326 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:33.326 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:33.326 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:33.326 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:25:33.327 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:33.327 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:33.327 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:33.327 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:25:33.327 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:33.328 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:33.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:33.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:33.329 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:33.329 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:33.329 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:33.330 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:33.330 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:33.330 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:33.330 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:33.331 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:33.331 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:33.331 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:33.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:33.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:33.332 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:33.332 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:33.332 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:33.332 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:33.333 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:33.333 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:33.333 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:33.333 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:33.334 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:33.334 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:33.334 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:33.334 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:33.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:33.335 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:33.335 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:33.335 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:33.335 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:33.336 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:33.337 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:25:38.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:38.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:38.340 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:38.384 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:38.384 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:38.438 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:38.438 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:38.438 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:38.438 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:38.439 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:38.439 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:38.439 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:38.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:38.440 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:38.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:38.440 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:38.440 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:38.441 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:38.441 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:38.441 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:38.441 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:38.442 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:38.442 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:38.442 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:38.442 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:38.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:38.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:38.443 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:38.443 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:38.444 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:38.444 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:25:38.444 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:38.444 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:38.445 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:38.445 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:25:38.445 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:38.445 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:38.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:38.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:38.446 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:38.446 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:38.447 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:38.447 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:38.447 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:38.447 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:38.448 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:38.448 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:38.448 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:38.448 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:38.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:38.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:38.449 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:38.449 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:38.449 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:38.450 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:38.450 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:38.450 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:38.450 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:38.451 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:38.451 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:38.451 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:38.451 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:38.452 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:38.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:38.452 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:38.452 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:38.453 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:38.453 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:38.453 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:38.454 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:25:43.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:43.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:43.457 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:43.505 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:43.505 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:43.561 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:43.562 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:43.562 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:43.562 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:43.562 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:43.563 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:43.563 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:43.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:43.563 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:43.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:43.564 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:43.564 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:43.564 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:43.564 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:43.565 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:43.565 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:43.565 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:43.565 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:43.566 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:43.566 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:43.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:43.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:43.566 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:43.567 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:43.567 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:43.567 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:25:43.567 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:43.568 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:43.568 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:43.568 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:25:43.568 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:43.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:43.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:43.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:43.569 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:43.569 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:43.569 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:43.570 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:43.570 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:43.570 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:43.570 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:43.571 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:43.571 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:43.571 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:43.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:43.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:43.572 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:43.572 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:43.572 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:43.572 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:43.573 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:43.573 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:43.573 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:43.573 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:43.573 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:43.574 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:43.574 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:43.574 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:43.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:43.574 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:43.575 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:43.575 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:43.575 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:43.575 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:43.576 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:25:48.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:48.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:48.580 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:48.624 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:48.625 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:48.685 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:48.685 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:48.685 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:48.685 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:48.686 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:48.686 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:48.686 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:48.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:48.687 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:48.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:48.687 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:48.687 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:48.687 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:48.688 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:48.688 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:48.688 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:48.688 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:48.689 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:48.689 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:48.689 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:48.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:48.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:48.690 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:48.690 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:48.690 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:48.690 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.091993_244.1", 2021-09-17T02:25:48.690 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:48.691 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:48.691 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:48.691 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.091993Z", 2021-09-17T02:25:48.691 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:48.692 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:48.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:48.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:48.692 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:48.692 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:48.693 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:48.693 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:48.693 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:48.693 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:48.693 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:48.694 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:48.694 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:48.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:48.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:48.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:48.695 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:48.695 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:48.695 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:48.695 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:48.696 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:48.696 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:48.696 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:48.696 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:48.696 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:48.697 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:48.697 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:48.697 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:48.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:48.698 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:48.698 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:48.698 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:48.698 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:48.698 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:48.700 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.091993_244.1 2021-09-17T02:25:53.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:53.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:53.702 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:53.745 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:53.746 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:53.799 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:53.800 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:53.800 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:53.800 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:53.801 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:53.801 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:53.801 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:53.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:53.802 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:53.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:53.802 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:53.802 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:53.802 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:53.803 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:53.803 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:53.803 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:53.803 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:53.804 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:53.804 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:53.804 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:53.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:53.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:53.805 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:53.805 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:53.805 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:53.805 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.157882_247.1", 2021-09-17T02:25:53.806 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:53.806 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:53.806 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:53.806 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.157882Z", 2021-09-17T02:25:53.807 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:53.807 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:53.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:53.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:53.807 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:53.808 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:53.808 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:53.808 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:53.808 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:53.809 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:53.809 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:53.809 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:53.809 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:53.810 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:53.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:53.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:53.811 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:53.811 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:53.811 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:53.811 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:53.811 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:53.812 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:53.812 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:53.812 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:53.812 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:53.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:53.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:53.813 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:53.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:53.813 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:53.814 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:53.814 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:53.814 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:53.814 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:53.815 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.157882_247.1 2021-09-17T02:25:58.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:58.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:25:58.818 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:25:58.862 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:25:58.863 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:25:58.916 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:25:58.916 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:25:58.917 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:25:58.917 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:25:58.917 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:25:58.917 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:25:58.917 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:58.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:58.918 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:25:58.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:58.918 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:25:58.919 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:58.919 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:58.919 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:25:58.919 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:58.919 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:25:58.920 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:58.920 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:25:58.920 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:58.920 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:58.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:58.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:58.921 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:25:58.921 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:58.921 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:58.922 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:25:58.922 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:58.922 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:58.922 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:58.923 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:25:58.923 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:58.923 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:58.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:58.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:58.924 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:25:58.924 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:58.924 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:58.924 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:58.925 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:58.925 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:58.925 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:58.925 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:58.925 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:58.926 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:58.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:58.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:25:58.926 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:25:58.927 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:25:58.927 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:25:58.927 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:25:58.927 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:25:58.927 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:25:58.928 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:25:58.928 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:25:58.928 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:25:58.928 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:58.928 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:58.929 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:25:58.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:25:58.929 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:25:58.929 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:25:58.930 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:25:58.930 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:25:58.930 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:25:58.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.067914_240.1 2021-09-17T02:26:03.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:03.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:03.934 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:03.980 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:03.980 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:04.035 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:26:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:26:04.035 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:26:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:26:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:26:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:26:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:04.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:26:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:26:04.037 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:04.038 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:26:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:26:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:04.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:26:04.040 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:26:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:04.041 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:26:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:04.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:04.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:04.043 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:26:04.043 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:04.043 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:04.044 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:04.044 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:04.044 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:04.044 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:04.045 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:04.045 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:04.045 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:04.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:04.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:04.046 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:26:04.046 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:04.046 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:04.046 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:04.047 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:04.047 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:04.047 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:04.047 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:04.048 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:04.048 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:04.048 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:04.048 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:26:04.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:04.049 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:26:04.049 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:26:04.049 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:26:04.049 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:04.050 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:26:04.051 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:26:09.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2021-09-17T02:26:09.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:09.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:09.056 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:09.106 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:09.107 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:09.149 INFO:teuthology.orchestra.run.smithi013.stderr:No --period given, using current period=3c964c05-2a9c-48af-9858-975bf7b57c1a 2021-09-17T02:26:09.163 INFO:teuthology.orchestra.run.smithi013.stdout:[ 2021-09-17T02:26:09.163 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:26:09.163 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:09.164 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:00:50.260780Z" 2021-09-17T02:26:09.164 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:26:09.164 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:26:09.164 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "1_1631845259.851939_401.1", 2021-09-17T02:26:09.165 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "2021-09-17T02:20:59.851939Z" 2021-09-17T02:26:09.165 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:26:09.165 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:26:09.165 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:26:09.165 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:26:09.166 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:26:09.166 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:26:09.166 INFO:teuthology.orchestra.run.smithi013.stdout: "marker": "", 2021-09-17T02:26:09.166 INFO:teuthology.orchestra.run.smithi013.stdout: "last_update": "0.000000" 2021-09-17T02:26:09.167 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:26:09.167 INFO:teuthology.orchestra.run.smithi013.stdout:] 2021-09-17T02:26:09.168 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T02:26:09.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:09.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:09.170 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:09.218 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:09.219 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:09.278 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:26:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:26:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:26:09.278 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:26:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:26:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:26:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:09.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:26:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:26:09.280 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:09.281 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:26:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:26:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:09.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:26:09.283 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.209532_249.1", 2021-09-17T02:26:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:09.284 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.209532Z", 2021-09-17T02:26:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:09.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:26:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:09.286 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:09.287 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:09.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:26:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:09.289 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:09.290 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:26:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:09.291 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:26:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:26:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:26:09.292 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:09.292 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:26:09.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.209532_249.1 2021-09-17T02:26:14.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:14.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:14.296 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:14.347 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:14.347 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:14.407 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:26:14.408 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:26:14.408 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:26:14.408 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:26:14.408 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:26:14.409 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:26:14.409 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:14.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:14.409 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:26:14.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:14.410 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:26:14.410 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:14.410 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:14.411 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:14.411 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:14.411 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:26:14.411 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:14.411 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:26:14.412 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:14.412 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:14.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:14.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:14.413 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:26:14.413 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:14.413 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:14.413 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.157882_247.1", 2021-09-17T02:26:14.414 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:14.414 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:14.414 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:14.414 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.157882Z", 2021-09-17T02:26:14.414 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:14.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:14.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:14.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:14.415 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:26:14.416 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:14.416 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:14.416 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:14.416 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:14.416 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:14.417 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:14.417 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:14.417 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:14.417 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:14.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:14.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:14.418 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:26:14.418 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:14.419 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:14.419 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:14.419 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:14.419 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:14.420 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:14.420 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:14.420 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:14.420 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:14.420 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:14.421 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:26:14.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:14.421 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:26:14.421 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:26:14.421 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:26:14.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:14.422 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:26:14.423 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.157882_247.1 2021-09-17T02:26:19.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:19.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:19.426 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:19.478 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:19.479 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:19.539 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:26:19.540 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:26:19.540 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:26:19.540 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:26:19.540 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:26:19.541 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:26:19.541 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:19.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:19.541 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:26:19.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:19.542 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:26:19.542 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:19.542 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:19.543 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:19.543 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:19.543 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:26:19.544 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:19.544 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:26:19.544 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:19.544 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:19.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:19.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:19.545 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:26:19.545 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:19.545 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:19.546 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.161433_248.1", 2021-09-17T02:26:19.546 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:19.546 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:19.546 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:19.546 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.161433Z", 2021-09-17T02:26:19.547 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:19.547 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:19.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:19.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:19.548 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:26:19.548 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:19.548 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:19.548 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:19.548 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:19.549 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:19.549 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:19.549 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:19.549 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:19.550 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:19.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:19.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:19.550 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:26:19.550 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:19.551 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:19.551 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:19.551 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:19.551 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:19.552 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:19.552 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:19.552 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:19.552 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:19.552 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:19.553 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:26:19.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:19.553 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:26:19.553 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:26:19.554 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:26:19.554 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:19.554 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:26:19.555 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.161433_248.1 2021-09-17T02:26:24.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:24.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:24.557 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:24.609 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:24.610 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:24.675 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:26:24.675 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:26:24.676 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:26:24.676 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:26:24.676 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:26:24.676 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:26:24.676 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:24.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:24.677 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:26:24.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:24.677 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:26:24.678 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:24.678 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:24.678 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:24.678 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:24.678 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:26:24.679 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:24.679 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:26:24.679 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:24.679 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:24.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:24.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:24.680 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:26:24.680 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:24.680 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:24.681 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.390428_266.1", 2021-09-17T02:26:24.681 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:24.681 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:24.681 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:24.681 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.390428Z", 2021-09-17T02:26:24.682 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:24.682 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:24.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:24.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:24.683 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:26:24.683 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:24.683 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:24.683 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:24.683 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:24.684 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:24.684 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:24.684 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:24.684 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:24.684 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:24.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:24.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:24.685 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:26:24.685 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:24.686 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:24.686 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:24.686 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:24.686 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:24.686 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:24.687 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:24.687 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:24.687 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:24.687 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:24.687 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:26:24.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:24.688 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:26:24.688 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:26:24.688 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:26:24.689 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:24.689 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:26:24.690 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.390428_266.1 2021-09-17T02:26:29.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:29.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:29.693 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:29.740 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:29.799 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:26:29.799 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:26:29.799 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:26:29.800 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:26:29.800 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:26:29.800 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:26:29.800 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:29.800 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:29.801 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:26:29.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:29.801 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:26:29.801 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:29.802 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:29.802 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:29.802 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:29.802 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:26:29.802 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:29.803 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:26:29.803 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:29.803 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:29.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:29.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:29.804 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:26:29.804 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:29.804 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:29.805 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:26:29.805 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:29.805 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:29.805 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:29.806 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:26:29.806 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:29.806 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:29.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:29.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:29.807 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:26:29.807 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:29.807 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:29.808 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:29.808 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:29.808 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:29.808 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:29.808 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:29.809 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:29.809 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:29.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:29.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:29.809 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:26:29.810 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:29.810 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:29.810 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:29.810 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:29.811 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:29.811 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:29.811 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:29.811 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:29.811 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:29.812 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:29.812 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:26:29.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:29.812 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:26:29.812 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:26:29.813 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:26:29.813 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:29.813 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:26:29.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.089761_243.1 2021-09-17T02:26:34.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:34.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:34.817 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:34.881 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:34.881 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:34.942 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:26:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:26:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:26:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:26:34.943 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:26:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:26:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:34.944 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:26:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:26:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:34.945 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:26:34.946 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:26:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:26:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:34.948 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:26:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:34.949 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:26:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:34.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:26:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:34.951 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:34.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:26:34.953 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:34.954 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:34.955 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:26:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:26:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:26:34.956 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:26:34.957 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:34.957 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:26:34.958 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.067914_240.1 2021-09-17T02:26:39.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:39.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:39.961 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:40.034 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:40.035 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:40.419 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:26:40.419 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:26:40.420 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:26:40.420 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:26:40.420 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:26:40.420 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:26:40.420 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:40.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:40.421 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:26:40.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:40.421 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:26:40.421 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:40.422 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:40.422 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:40.422 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:40.422 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:26:40.423 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:40.423 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:26:40.423 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:40.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:40.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:40.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:40.424 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:26:40.424 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:40.424 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:40.424 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:26:40.425 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:40.425 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:40.425 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:40.425 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:26:40.426 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:40.426 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:40.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:40.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:40.426 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:26:40.427 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:40.427 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:40.427 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:40.427 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:40.427 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:40.428 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:40.428 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:40.428 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:40.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:40.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:40.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:40.429 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:26:40.429 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:40.429 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:40.430 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:40.430 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:40.430 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:40.430 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:40.430 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:40.431 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:40.431 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:40.431 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:40.431 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:26:40.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:40.432 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:26:40.432 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:26:40.432 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:26:40.432 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:40.433 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:26:40.434 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.067914_240.1 2021-09-17T02:26:45.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:45.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:45.437 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:45.515 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:45.516 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:45.603 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:26:45.877 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:26:45.877 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:26:45.877 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:26:45.878 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:26:45.878 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:26:45.878 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:45.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:45.879 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:26:45.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:45.879 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:26:45.879 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:45.879 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:45.880 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:45.880 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:45.880 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:26:45.880 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:45.880 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:26:45.881 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:45.881 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:45.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:45.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:45.882 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:26:45.882 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:45.882 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:45.882 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.268651_252.1", 2021-09-17T02:26:45.883 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:45.883 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:45.883 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:45.883 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.268651Z", 2021-09-17T02:26:45.883 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:45.884 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:45.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:45.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:45.884 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:26:45.885 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:45.885 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:45.885 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:45.885 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:45.885 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:45.886 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:45.886 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:45.886 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:45.886 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:45.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:45.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:45.887 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:26:45.887 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:45.887 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:45.888 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:45.888 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:45.888 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:45.888 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:45.888 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:45.889 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:45.889 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:45.889 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:45.889 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:26:45.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:45.890 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:26:45.890 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:26:45.890 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:26:45.891 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:45.891 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:26:45.892 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.268651_252.1 2021-09-17T02:26:50.893 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:50.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:50.895 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:51.173 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:26:51.174 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:26:51.174 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:26:51.174 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:26:51.174 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:26:51.175 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:26:51.175 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:51.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:51.175 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:26:51.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:51.176 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:26:51.176 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:51.176 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:51.176 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:51.177 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:51.177 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:26:51.177 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:51.177 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:26:51.177 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:51.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:51.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:51.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:51.178 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:26:51.179 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:51.179 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:51.179 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.157882_247.1", 2021-09-17T02:26:51.179 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:51.180 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:51.180 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:51.180 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.157882Z", 2021-09-17T02:26:51.180 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:51.180 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:51.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:51.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:51.181 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:26:51.181 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:51.181 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:51.182 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:51.182 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:51.182 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:51.182 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:51.182 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:51.183 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:51.183 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:51.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:51.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:51.183 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:26:51.184 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:51.184 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:51.184 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:51.184 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:51.185 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:51.185 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:51.185 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:51.185 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:51.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:51.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:51.186 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:26:51.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:51.187 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:26:51.187 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:26:51.188 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:26:51.188 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:51.189 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:26:51.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.157882_247.1 2021-09-17T02:26:56.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:56.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:26:56.194 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:26:56.238 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:26:56.238 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:26:56.291 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:26:56.291 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:26:56.291 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:26:56.291 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:26:56.292 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:26:56.292 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:26:56.292 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:56.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:56.293 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:26:56.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:56.293 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:26:56.293 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:56.293 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:56.294 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:26:56.294 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:56.294 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:26:56.294 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:56.294 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:26:56.295 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:56.295 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:56.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:56.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:56.296 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:26:56.296 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:56.296 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:56.296 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:26:56.297 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:56.297 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:56.297 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:56.297 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:26:56.298 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:56.298 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:56.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:56.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:56.298 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:26:56.299 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:56.299 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:56.299 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:56.299 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:56.299 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:56.300 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:56.300 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:56.300 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:56.300 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:56.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:56.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:26:56.301 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:26:56.301 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:26:56.302 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:26:56.302 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:26:56.302 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:26:56.302 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:26:56.302 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:26:56.303 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:26:56.303 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:26:56.303 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:56.303 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:56.303 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:26:56.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:26:56.304 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:26:56.304 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:26:56.304 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:26:56.305 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:26:56.305 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:26:56.306 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.088776_242.1 2021-09-17T02:27:01.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:01.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:01.308 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:01.352 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:01.352 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:01.406 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:01.406 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:01.407 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:01.408 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:01.409 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:01.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:01.411 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:27:01.412 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:01.412 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:01.412 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:01.412 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:27:01.413 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:01.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:01.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:01.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:01.413 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:01.414 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:01.414 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:01.414 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:01.414 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:01.415 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:01.415 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:01.415 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:01.415 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:01.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:01.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:01.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:01.416 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:01.416 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:01.416 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:01.417 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:01.417 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:01.417 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:01.417 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:01.418 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:01.418 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:01.418 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:01.418 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:01.418 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:01.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:01.419 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:01.419 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:01.419 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:01.420 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:01.420 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:01.421 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.086132_241.1 2021-09-17T02:27:06.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:06.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:06.424 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:06.468 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:06.468 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:06.523 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:06.524 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:06.525 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:06.526 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:06.527 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:06.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:27:06.529 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:27:06.530 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:06.531 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:06.532 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:06.533 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:06.534 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:06.535 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:06.535 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:06.535 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:06.535 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:06.536 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:06.536 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:06.536 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:06.536 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:06.536 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:06.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:06.537 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:06.537 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:06.537 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:06.538 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:06.538 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:06.539 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.089761_243.1 2021-09-17T02:27:11.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:11.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:11.542 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:11.586 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:11.586 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:11.642 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:11.642 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:11.642 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:11.642 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:11.643 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:11.643 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:11.643 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:11.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:11.644 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:11.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:11.644 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:11.644 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:11.645 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:11.645 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:11.645 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:11.645 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:11.645 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:11.646 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:11.646 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:11.646 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:11.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:11.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:11.647 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:11.647 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:11.647 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:11.648 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:27:11.648 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:11.648 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:11.648 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:11.649 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:27:11.649 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:11.649 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:11.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:11.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:11.650 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:11.650 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:11.650 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:11.650 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:11.651 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:11.651 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:11.651 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:11.651 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:11.652 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:11.652 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:11.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:11.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:11.653 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:11.653 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:11.653 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:11.653 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:11.653 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:11.654 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:11.655 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:11.655 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:11.655 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:11.655 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:11.656 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:11.656 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:11.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:11.656 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:11.657 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:11.657 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:11.657 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:11.657 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:11.658 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:27:16.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:16.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:16.662 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:16.709 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:16.709 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:16.765 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:16.765 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:16.766 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:16.766 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:16.766 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:16.766 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:16.767 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:16.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:16.767 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:16.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:16.767 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:16.768 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:16.768 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:16.768 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:16.768 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:16.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:16.770 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.268651_252.1", 2021-09-17T02:27:16.771 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.268651Z", 2021-09-17T02:27:16.772 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:16.773 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:16.774 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:16.775 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:16.776 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:16.777 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:16.778 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:16.779 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:16.780 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:16.781 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:16.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.268651_252.1 2021-09-17T02:27:21.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:21.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:21.785 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:21.830 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:21.830 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:21.884 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:21.884 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:21.885 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:21.885 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:21.885 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:21.885 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:21.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:21.886 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:21.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:21.886 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:21.886 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:21.887 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:21.887 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:21.887 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:21.887 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:21.887 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:21.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:21.889 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:27:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:21.890 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:27:21.891 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:21.891 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:21.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:21.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:21.892 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:21.892 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:21.892 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:21.892 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:21.892 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:21.893 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:21.894 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:21.895 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:21.896 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:21.897 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:21.897 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:21.897 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:21.897 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:21.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:21.898 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:21.900 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:21.900 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:21.902 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.067914_240.1 2021-09-17T02:27:26.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:26.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:26.905 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:26.952 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:26.952 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:27.007 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:27.007 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:27.008 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:27.008 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:27.008 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:27.008 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:27.009 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:27.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:27.009 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:27.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:27.010 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:27.010 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:27.010 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:27.010 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:27.010 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:27.011 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:27.011 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:27.011 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:27.011 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:27.012 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:27.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:27.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:27.012 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:27.012 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:27.013 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:27.013 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:27:27.013 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:27.013 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:27.014 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:27.014 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:27:27.014 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:27.014 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:27.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:27.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:27.015 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:27.015 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:27.015 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:27.016 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:27.016 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:27.016 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:27.016 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:27.016 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:27.017 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:27.017 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:27.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:27.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:27.018 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:27.018 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:27.018 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:27.018 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:27.019 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:27.019 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:27.019 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:27.019 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:27.019 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:27.020 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:27.020 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:27.020 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:27.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:27.020 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:27.021 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:27.021 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:27.021 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:27.021 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:27.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.089761_243.1 2021-09-17T02:27:32.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:32.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:32.025 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:32.070 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:32.070 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:32.124 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:32.124 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:32.124 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:32.125 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:32.125 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:32.125 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:32.125 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:32.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:32.126 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:32.126 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:32.126 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:32.126 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:32.127 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:32.127 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:32.127 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:32.127 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:32.128 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:32.128 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:32.128 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:32.128 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:32.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:32.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:32.129 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:32.129 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:32.129 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:32.130 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:27:32.130 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:32.130 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:32.131 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:32.131 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:27:32.131 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:32.131 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:32.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:32.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:32.132 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:32.132 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:32.133 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:32.133 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:32.133 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:32.133 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:32.134 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:32.134 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:32.134 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:32.134 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:32.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:32.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:32.135 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:32.135 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:32.135 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:32.136 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:32.136 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:32.136 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:32.136 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:32.137 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:32.137 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:32.137 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:32.137 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:32.137 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:32.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:32.138 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:32.138 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:32.139 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:32.139 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:32.139 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:32.140 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.067914_240.1 2021-09-17T02:27:37.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:37.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:37.143 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:37.194 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:37.195 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:37.256 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:37.256 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:37.256 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:37.257 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:37.257 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:37.257 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:37.257 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:37.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:37.258 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:37.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:37.258 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:37.259 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:37.259 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:37.259 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:37.259 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:37.260 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:37.260 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:37.260 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:37.260 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:37.261 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:37.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:37.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:37.261 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:37.262 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:37.262 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:37.262 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.103309_245.1", 2021-09-17T02:27:37.262 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:37.263 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:37.263 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:37.263 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.103309Z", 2021-09-17T02:27:37.263 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:37.264 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:37.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:37.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:37.264 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:37.265 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:37.265 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:37.265 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:37.265 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:37.266 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:37.266 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:37.266 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:37.266 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:37.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:37.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:37.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:37.267 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:37.267 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:37.268 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:37.268 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:37.268 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:37.268 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:37.269 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:37.269 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:37.269 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:37.269 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:37.269 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:37.270 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:37.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:37.270 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:37.270 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:37.271 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:37.271 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:37.271 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:37.272 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.103309_245.1 2021-09-17T02:27:42.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:42.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:42.275 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:42.327 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:42.328 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:42.405 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:42.405 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:42.406 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:42.406 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:42.406 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:42.407 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:42.407 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:42.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:42.407 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:42.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:42.408 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:42.408 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:42.408 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:42.408 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:42.409 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:42.409 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:42.409 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:42.409 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:42.409 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:42.410 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:42.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:42.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:42.410 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:42.411 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:42.411 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:42.411 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.272193_254.1", 2021-09-17T02:27:42.411 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:42.411 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:42.412 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:42.412 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.272193Z", 2021-09-17T02:27:42.412 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:42.412 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:42.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:42.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:42.413 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:42.413 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:42.413 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:42.414 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:42.414 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:42.414 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:42.414 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:42.415 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:42.415 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:42.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:42.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:42.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:42.416 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:42.416 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:42.416 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:42.416 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:42.417 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:42.417 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:42.417 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:42.417 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:42.417 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:42.418 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:42.418 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:42.418 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:42.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:42.418 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:42.419 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:42.419 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:42.419 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:42.419 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:42.421 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.272193_254.1 2021-09-17T02:27:47.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:47.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:47.423 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:47.500 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:47.501 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:47.841 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:47.841 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:47.841 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:47.842 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:47.842 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:47.842 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:47.842 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:47.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:47.843 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:47.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:47.844 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:47.844 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:47.844 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:47.844 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:47.844 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:47.845 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:47.845 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:47.845 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:47.845 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:47.845 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:47.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:47.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:47.846 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:47.846 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:47.847 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:47.847 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.287190_256.1", 2021-09-17T02:27:47.847 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:47.847 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:47.847 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:47.848 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.287190Z", 2021-09-17T02:27:47.848 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:47.848 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:47.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:47.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:47.849 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:47.849 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:47.849 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:47.849 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:47.850 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:47.850 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:47.850 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:47.850 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:47.850 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:47.851 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:47.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:47.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:47.851 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:47.851 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:47.852 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:47.852 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:47.852 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:47.852 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:47.853 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:47.853 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:47.853 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:47.853 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:47.853 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:47.854 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:47.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:47.854 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:47.854 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:47.854 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:47.855 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:47.855 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:47.856 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.287190_256.1 2021-09-17T02:27:52.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:52.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:52.859 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:52.905 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:52.906 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:52.962 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:52.962 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:52.963 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:52.963 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:52.963 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:52.963 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:52.964 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:52.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:52.964 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:52.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:52.964 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:52.965 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:52.965 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:52.965 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:52.965 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:52.966 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:52.966 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:52.966 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:52.966 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:52.966 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:52.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:52.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:52.967 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:52.967 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:52.968 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:52.968 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.091993_244.1", 2021-09-17T02:27:52.968 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:52.968 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:52.968 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:52.969 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.091993Z", 2021-09-17T02:27:52.969 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:52.969 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:52.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:52.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:52.970 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:52.970 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:52.970 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:52.970 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:52.971 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:52.972 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:52.973 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:52.973 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:52.973 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:52.973 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:52.974 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:52.975 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:52.975 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:52.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:52.975 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:52.976 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:52.976 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:52.976 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:52.976 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:52.977 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.091993_244.1 2021-09-17T02:27:57.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:57.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:27:57.980 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:27:58.065 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:27:58.066 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:27:58.317 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:27:58.317 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:27:58.318 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:27:58.318 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:27:58.318 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:27:58.319 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:27:58.319 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:58.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:58.319 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:27:58.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:58.320 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:27:58.320 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:58.320 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:58.321 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:27:58.321 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:58.321 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:27:58.321 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:58.321 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:27:58.322 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:58.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:58.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:58.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:58.323 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:27:58.323 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:58.323 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:58.323 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.212386_250.1", 2021-09-17T02:27:58.324 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:58.324 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:58.324 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:58.324 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.212386Z", 2021-09-17T02:27:58.324 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:58.325 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:58.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:58.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:58.325 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:27:58.326 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:58.326 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:58.326 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:58.326 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:58.326 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:58.327 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:58.327 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:58.327 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:58.327 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:58.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:58.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:27:58.328 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:27:58.328 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:27:58.328 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:27:58.329 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:27:58.329 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:27:58.329 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:27:58.329 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:27:58.330 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:27:58.330 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:27:58.330 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:58.330 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:58.330 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:27:58.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:27:58.331 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:27:58.331 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:27:58.331 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:27:58.332 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:27:58.332 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:27:58.333 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.212386_250.1 2021-09-17T02:28:03.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:03.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:03.335 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:03.413 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:03.413 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:28:03.605 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:28:03.605 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:28:03.606 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:28:03.606 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:28:03.606 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:28:03.606 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:28:03.607 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:03.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:03.607 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:28:03.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:03.608 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:28:03.608 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:03.608 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:03.608 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:28:03.609 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:03.609 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:28:03.609 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:03.609 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:28:03.610 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:03.610 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:03.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:03.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:03.611 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:28:03.611 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:03.611 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:03.611 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:28:03.611 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:03.612 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:03.612 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:03.612 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:28:03.612 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:03.613 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:03.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:03.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:03.613 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:28:03.614 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:03.614 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:03.614 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:03.614 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:03.615 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:03.615 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:03.615 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:03.615 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:03.615 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:03.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:03.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:03.616 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:28:03.616 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:03.616 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:03.617 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:03.617 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:03.617 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:03.617 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:03.617 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:03.618 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:03.618 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:03.618 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:03.618 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:28:03.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:03.619 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:28:03.619 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:28:03.619 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:28:03.620 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:03.620 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:28:03.622 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.089761_243.1 2021-09-17T02:28:08.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:08.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:08.625 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:08.688 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:08.688 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:28:08.741 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:28:08.742 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:28:08.742 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:28:08.742 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:28:08.742 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:28:08.743 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:28:08.743 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:08.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:08.743 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:28:08.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:08.744 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:28:08.744 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:08.744 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:08.744 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:28:08.745 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:08.745 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:28:08.745 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:08.745 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:28:08.746 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:08.746 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:08.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:08.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:08.746 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:28:08.747 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:08.747 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:08.747 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.105759_246.1", 2021-09-17T02:28:08.747 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:08.748 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:08.748 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:08.748 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.105759Z", 2021-09-17T02:28:08.748 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:08.748 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:08.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:08.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:08.749 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:28:08.749 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:08.750 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:08.750 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:08.750 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:08.750 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:08.751 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:08.751 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:08.751 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:08.751 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:08.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:08.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:08.752 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:28:08.752 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:08.752 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:08.753 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:08.753 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:08.753 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:08.753 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:08.754 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:08.754 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:08.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:08.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:08.755 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:28:08.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:08.755 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:28:08.755 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:28:08.756 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:28:08.756 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:08.756 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:28:08.757 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.105759_246.1 2021-09-17T02:28:13.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:13.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:13.760 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:13.804 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:13.804 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:28:13.856 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:28:13.856 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:28:13.856 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:28:13.857 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:28:13.857 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:28:13.857 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:28:13.857 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:13.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:13.858 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:28:13.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:13.858 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:28:13.858 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:13.859 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:13.859 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:28:13.859 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:13.859 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:28:13.860 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:13.860 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:28:13.860 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:13.860 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:13.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:13.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:13.861 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:28:13.861 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:13.862 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:13.862 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:28:13.862 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:13.862 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:13.862 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:13.863 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:28:13.863 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:13.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:13.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:13.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:13.864 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:28:13.864 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:13.864 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:13.864 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:13.865 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:13.865 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:13.865 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:13.865 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:13.866 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:13.866 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:13.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:13.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:13.866 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:28:13.867 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:13.867 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:13.867 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:13.867 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:13.867 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:13.868 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:13.868 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:13.868 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:13.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:13.869 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:13.869 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:28:13.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:13.869 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:28:13.869 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:28:13.870 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:28:13.870 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:13.870 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:28:13.871 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.086132_241.1 2021-09-17T02:28:18.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:18.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:18.873 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:18.924 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:18.925 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:28:18.979 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:28:18.979 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:28:18.979 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:28:18.979 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:28:18.980 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:28:18.980 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:28:18.980 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:18.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:18.980 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:28:18.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:18.981 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:28:18.981 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:18.981 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:18.981 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:28:18.982 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:18.982 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:28:18.982 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:18.982 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:28:18.983 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:18.983 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:18.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:18.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:18.983 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:28:18.984 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:18.984 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:18.984 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.091993_244.1", 2021-09-17T02:28:18.984 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:18.985 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:18.985 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:18.985 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.091993Z", 2021-09-17T02:28:18.985 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:18.985 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:18.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:18.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:18.986 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:28:18.986 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:18.987 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:18.987 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:18.987 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:18.987 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:18.987 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:18.988 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:18.988 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:18.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:18.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:18.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:18.989 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:28:18.989 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:18.989 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:18.989 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:18.990 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:18.990 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:18.990 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:18.990 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:18.990 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:18.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:18.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:18.991 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:28:18.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:18.992 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:28:18.992 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:28:18.992 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:28:18.992 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:18.992 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:28:18.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.091993_244.1 2021-09-17T02:28:23.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:23.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:23.996 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:24.042 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:28:24.098 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:28:24.099 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:28:24.099 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:28:24.100 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:28:24.100 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:28:24.100 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:28:24.101 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:24.101 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:24.102 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:28:24.102 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:24.103 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:28:24.103 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:24.103 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:24.103 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:28:24.104 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:24.104 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:28:24.104 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:24.104 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:28:24.105 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:24.105 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:24.105 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:24.105 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:24.105 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:28:24.106 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:24.106 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:24.106 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.105759_246.1", 2021-09-17T02:28:24.106 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:24.107 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:24.107 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:24.107 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.105759Z", 2021-09-17T02:28:24.108 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:24.108 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:24.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:24.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:24.108 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:28:24.109 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:24.109 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:24.109 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:24.109 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:24.110 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:24.110 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:24.111 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:24.111 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:24.111 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:24.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:24.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:24.112 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:28:24.112 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:24.112 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:24.112 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:24.113 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:24.113 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:24.113 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:24.113 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:24.114 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:24.114 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:24.115 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:24.115 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:28:24.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:24.116 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:28:24.116 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:28:24.117 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:28:24.117 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:24.118 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:28:24.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.105759_246.1 2021-09-17T02:28:29.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:29.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:29.121 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:28:29.234 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:28:29.234 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:28:29.234 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:28:29.235 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:28:29.235 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:28:29.235 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:28:29.235 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:29.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:29.236 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:28:29.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:29.236 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:28:29.236 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:29.237 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:29.237 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:28:29.237 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:29.237 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:28:29.238 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:29.238 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:28:29.238 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:29.238 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:29.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:29.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:29.239 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:28:29.239 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:29.239 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:29.240 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:28:29.240 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:29.240 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:29.240 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:29.240 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:28:29.241 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:29.241 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:29.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:29.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:29.242 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:28:29.242 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:29.242 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:29.242 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:29.242 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:29.243 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:29.243 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:29.243 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:29.243 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:29.244 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:29.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:29.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:29.244 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:28:29.244 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:29.245 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:29.245 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:29.245 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:29.245 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:29.245 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:29.246 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:29.246 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:29.246 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:29.246 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:29.247 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:28:29.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:29.247 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:28:29.247 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:28:29.247 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:28:29.248 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:29.248 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:28:29.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:28:34.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:34.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:34.251 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:34.303 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:34.303 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:28:34.363 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:28:34.364 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:28:34.364 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:28:34.364 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:28:34.364 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:28:34.365 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:28:34.365 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:34.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:34.365 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:28:34.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:34.366 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:28:34.366 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:34.366 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:34.366 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:28:34.367 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:34.367 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:28:34.367 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:34.367 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:28:34.368 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:34.368 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:34.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:34.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:34.368 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:28:34.369 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:34.369 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:34.369 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.157882_247.1", 2021-09-17T02:28:34.369 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:34.369 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:34.370 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:34.370 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.157882Z", 2021-09-17T02:28:34.370 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:34.370 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:34.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:34.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:34.371 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:28:34.371 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:34.371 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:34.372 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:34.372 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:34.372 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:34.372 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:34.372 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:34.373 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:34.373 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:34.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:34.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:34.374 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:28:34.374 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:34.374 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:34.374 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:34.374 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:34.375 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:34.375 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:34.375 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:34.375 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:34.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:34.376 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:34.376 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:28:34.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:34.376 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:28:34.376 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:28:34.377 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:28:34.377 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:34.377 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:28:34.378 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.157882_247.1 2021-09-17T02:28:39.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:39.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:39.381 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:39.429 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:39.429 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:28:39.487 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:28:39.487 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:28:39.488 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:28:39.488 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:28:39.488 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:28:39.488 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:28:39.489 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:39.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:39.489 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:28:39.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:39.490 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:28:39.490 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:39.490 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:39.490 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:28:39.490 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:39.491 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:28:39.491 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:39.491 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:28:39.491 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:39.492 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:39.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:39.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:39.492 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:28:39.492 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:39.493 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:39.493 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.067914_240.1", 2021-09-17T02:28:39.493 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:39.493 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:39.493 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:39.494 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.067914Z", 2021-09-17T02:28:39.494 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:39.494 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:39.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:39.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:39.495 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:28:39.495 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:39.496 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:39.496 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:39.496 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:39.496 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:39.496 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:39.497 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:39.497 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:39.497 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:39.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:39.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:39.498 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:28:39.498 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:39.498 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:39.499 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:39.499 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:39.499 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:39.499 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:39.499 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:39.500 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:39.500 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:39.500 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:39.500 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:28:39.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:39.501 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:28:39.501 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:28:39.501 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:28:39.501 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:39.502 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:28:39.503 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.067914_240.1 2021-09-17T02:28:44.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:44.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:44.505 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:44.554 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:44.555 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:28:44.618 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:28:44.618 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:28:44.618 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:28:44.619 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:28:44.619 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:28:44.619 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:28:44.619 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:44.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:44.620 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:28:44.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:44.620 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:28:44.620 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:44.621 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:44.621 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:28:44.621 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:44.621 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:28:44.622 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:44.622 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:28:44.622 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:44.622 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:44.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:44.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:44.623 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:28:44.623 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:44.623 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:44.624 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:28:44.624 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:44.624 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:44.624 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:44.625 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:28:44.625 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:44.625 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:44.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:44.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:44.626 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:28:44.626 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:44.626 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:44.626 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:44.627 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:44.627 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:44.627 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:44.627 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:44.627 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:44.628 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:44.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:44.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:44.628 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:28:44.629 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:44.629 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:44.629 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:44.629 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:44.630 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:44.630 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:44.630 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:44.630 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:44.631 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:44.631 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:44.631 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:28:44.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:44.631 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:28:44.632 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:28:44.632 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:28:44.632 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:44.633 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:28:44.634 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:28:49.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:49.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:49.636 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:49.687 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:49.688 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:28:49.747 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:28:49.747 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:28:49.748 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:28:49.748 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:28:49.748 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:28:49.748 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:28:49.749 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:49.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:49.749 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:28:49.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:49.750 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:28:49.750 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:49.750 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:49.750 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:28:49.751 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:49.751 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:28:49.751 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:49.751 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:28:49.752 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:49.752 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:49.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:49.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:49.753 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:28:49.753 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:49.753 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:49.754 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.089761_243.1", 2021-09-17T02:28:49.754 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:49.754 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:49.754 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:49.755 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.089761Z", 2021-09-17T02:28:49.755 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:49.755 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:49.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:49.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:49.756 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:28:49.756 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:49.756 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:49.757 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:49.757 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:49.758 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:49.758 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:49.758 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:49.758 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:49.759 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:49.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:49.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:49.760 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:28:49.760 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:49.760 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:49.760 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:49.761 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:49.761 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:49.761 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:49.761 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:49.762 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:49.762 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:49.762 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:49.762 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:28:49.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:49.763 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:28:49.763 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:28:49.763 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:28:49.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:49.764 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:28:49.765 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.089761_243.1 2021-09-17T02:28:54.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:54.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:54.768 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:54.817 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:54.818 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:28:54.877 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:28:54.877 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:28:54.878 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:28:54.878 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:28:54.878 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:28:54.878 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:28:54.879 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:54.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:54.879 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:28:54.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:54.880 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:28:54.880 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:54.880 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:54.880 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:28:54.881 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:54.881 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:28:54.881 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:54.881 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:28:54.882 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:54.882 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:54.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:54.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:54.883 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:28:54.883 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:54.883 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:54.884 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.209532_249.1", 2021-09-17T02:28:54.884 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:54.884 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:54.884 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.209532Z", 2021-09-17T02:28:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:54.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:54.886 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:28:54.886 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:54.887 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:54.887 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:54.887 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:54.887 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:54.888 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:54.888 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:54.888 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:54.888 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:54.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:54.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:28:54.889 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:28:54.890 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:28:54.890 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:28:54.890 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:28:54.890 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:28:54.891 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:28:54.891 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:28:54.891 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:28:54.891 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:28:54.892 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:54.892 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:54.892 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:28:54.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:28:54.893 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:28:54.893 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:28:54.893 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:28:54.893 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:28:54.894 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:28:54.895 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.209532_249.1 2021-09-17T02:28:59.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:59.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:28:59.898 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:28:59.947 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:28:59.948 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:00.011 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:00.011 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:00.012 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:00.012 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:00.012 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:00.012 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:00.013 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:00.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:00.013 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:00.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:00.014 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:00.014 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:00.014 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:00.014 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:00.014 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:00.015 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:00.015 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:00.015 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:00.015 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:00.016 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:00.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:00.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:00.016 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:00.016 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:00.017 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:00.017 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:29:00.017 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:00.017 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:00.018 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:00.018 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:29:00.018 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:00.018 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:00.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:00.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:00.019 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:00.019 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:00.020 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:00.020 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:00.020 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:00.020 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:00.020 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:00.021 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:00.021 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:00.021 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:00.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:00.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:00.022 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:00.022 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:00.022 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:00.022 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:00.023 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:00.023 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:00.023 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:00.023 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:00.023 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:00.024 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:00.024 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:00.024 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:00.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:00.025 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:00.025 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:00.025 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:00.025 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:00.025 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:00.026 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.086132_241.1 2021-09-17T02:29:05.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:05.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:05.029 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:29:05.082 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:29:05.083 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:05.144 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:05.144 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:05.144 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:05.145 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:05.145 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:05.145 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:05.145 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:05.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:05.146 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:05.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:05.146 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:05.147 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:05.147 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:05.148 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:05.148 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:05.148 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:05.148 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:05.149 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:05.149 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:05.149 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:05.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:05.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:05.150 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:05.150 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:05.150 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:05.150 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:29:05.151 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:05.151 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:05.151 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:05.151 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:29:05.151 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:05.152 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:05.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:05.152 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:05.152 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:05.152 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:05.153 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:05.153 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:05.153 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:05.153 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:05.154 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:05.154 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:05.154 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:05.154 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:05.154 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:05.155 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:05.155 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:05.155 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:05.155 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:05.156 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:05.156 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:05.156 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:05.156 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:05.156 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:05.157 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:05.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:05.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:05.157 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:05.158 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:05.158 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:05.158 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:05.158 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:05.159 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:05.159 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:05.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.088776_242.1 2021-09-17T02:29:10.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:10.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:10.162 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:29:10.212 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:29:10.213 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:10.317 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:10.318 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:10.319 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:10.319 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:10.319 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:10.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:10.320 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:10.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:10.320 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:10.320 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:10.320 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:10.321 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:10.321 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:10.321 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:10.321 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:10.322 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:10.322 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:10.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:10.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:10.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:10.323 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:10.323 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:10.323 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:10.323 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:29:10.324 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:10.324 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:10.324 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:10.324 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:29:10.324 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:10.325 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:10.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:10.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:10.326 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:10.326 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:10.326 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:10.326 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:10.326 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:10.327 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:10.327 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:10.327 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:10.327 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:10.328 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:10.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:10.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:10.328 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:10.329 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:10.329 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:10.329 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:10.329 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:10.329 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:10.330 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:10.330 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:10.330 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:10.330 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:10.331 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:10.331 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:10.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:10.331 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:10.331 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:10.332 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:10.332 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:10.332 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:10.333 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.086132_241.1 2021-09-17T02:29:15.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:15.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:15.336 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:29:15.387 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:29:15.387 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:15.580 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:15.581 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:15.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:15.583 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:15.584 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:15.584 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:15.584 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:15.584 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:15.585 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:15.585 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:15.585 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:15.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:15.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:15.586 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:15.586 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:15.586 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:15.586 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.288419_257.1", 2021-09-17T02:29:15.587 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:15.587 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:15.587 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:15.587 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.288419Z", 2021-09-17T02:29:15.587 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:15.588 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:15.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:15.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:15.588 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:15.589 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:15.589 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:15.589 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:15.589 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:15.589 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:15.590 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:15.590 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:15.590 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:15.590 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:15.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:15.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:15.591 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:15.591 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:15.591 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:15.592 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:15.592 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:15.592 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:15.592 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:15.592 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:15.593 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:15.593 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:15.593 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:15.593 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:15.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:15.594 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:15.594 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:15.594 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:15.594 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:15.595 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:15.595 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.288419_257.1 2021-09-17T02:29:20.596 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:20.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:20.598 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:29:20.689 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:29:20.690 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:20.966 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:20.967 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:20.967 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:20.967 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:20.967 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:20.967 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:20.968 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:20.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:20.968 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:20.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:20.969 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:20.969 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:20.969 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:20.969 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:20.969 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:20.970 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:20.970 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:20.970 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:20.970 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:20.971 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:20.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:20.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:20.971 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:20.972 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:20.972 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:20.972 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:29:20.972 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:20.972 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:20.973 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:20.973 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:29:20.973 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:20.973 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:20.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:20.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:20.974 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:20.974 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:20.974 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:20.975 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:20.975 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:20.975 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:20.975 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:20.975 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:20.976 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:20.976 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:20.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:20.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:20.977 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:20.977 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:20.977 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:20.977 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:20.977 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:20.978 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:20.978 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:20.978 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:20.978 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:20.978 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:20.979 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:20.979 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:20.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:20.979 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:20.980 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:20.980 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:20.980 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:20.980 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:20.981 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:29:25.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:25.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:25.984 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:29:26.040 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:29:26.041 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:26.143 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:26.143 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:26.143 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:26.143 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:26.144 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:26.144 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:26.144 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:26.144 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:26.145 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:26.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:26.145 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:26.145 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:26.146 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:26.146 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:26.146 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:26.146 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:26.147 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:26.147 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:26.147 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:26.147 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:26.148 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:26.148 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:26.148 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:26.148 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:26.148 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:26.149 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:29:26.149 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:26.149 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:26.149 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:26.150 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:29:26.150 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:26.150 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:26.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:26.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:26.151 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:26.151 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:26.151 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:26.151 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:26.152 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:26.152 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:26.152 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:26.152 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:26.153 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:26.153 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:26.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:26.153 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:26.154 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:26.154 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:26.154 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:26.154 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:26.155 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:26.155 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:26.155 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:26.155 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:26.155 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:26.156 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:26.156 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:26.156 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:26.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:26.157 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:26.157 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:26.157 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:26.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:26.158 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:26.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:29:31.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:31.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:31.162 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:29:31.221 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:29:31.222 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:31.408 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:31.408 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:31.409 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:31.409 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:31.409 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:31.409 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:31.410 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:31.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:31.410 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:31.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:31.410 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:31.411 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:31.411 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:31.411 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:31.411 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:31.412 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:31.412 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:31.412 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:31.412 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:31.412 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:31.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:31.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:31.413 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:31.413 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:31.414 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:31.414 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:29:31.414 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:31.414 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:31.414 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:31.415 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:29:31.415 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:31.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:31.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:31.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:31.416 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:31.416 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:31.416 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:31.416 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:31.416 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:31.417 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:31.417 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:31.417 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:31.417 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:31.417 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:31.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:31.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:31.418 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:31.418 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:31.419 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:31.419 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:31.419 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:31.419 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:31.419 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:31.420 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:31.420 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:31.420 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:31.420 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:31.421 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:31.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:31.421 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:31.421 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:31.421 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:31.422 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:31.422 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:31.422 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.088776_242.1 2021-09-17T02:29:36.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:36.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:36.426 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:29:36.501 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:29:36.502 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:36.753 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:36.753 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:36.754 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:36.754 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:36.754 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:36.754 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:36.755 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:36.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:36.755 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:36.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:36.756 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:36.756 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:36.756 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:36.756 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:36.757 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:36.757 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:36.757 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:36.758 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:36.758 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:36.758 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:36.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:36.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:36.759 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:36.759 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:36.759 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:36.760 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:29:36.760 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:36.760 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:36.760 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:36.761 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:29:36.761 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:36.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:36.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:36.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:36.762 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:36.762 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:36.762 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:36.762 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:36.763 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:36.763 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:36.763 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:36.763 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:36.764 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:36.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:36.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:36.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:36.764 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:36.765 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:36.765 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:36.765 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:36.765 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:36.766 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:36.766 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:36.766 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:36.766 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:36.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:36.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:36.767 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:36.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:36.767 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:36.768 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:36.768 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:36.768 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:36.768 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:36.769 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.088776_242.1 2021-09-17T02:29:41.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:41.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:41.772 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:29:41.872 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:29:41.873 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:42.035 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:42.035 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:42.035 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:42.036 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:42.036 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:42.036 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:42.036 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:42.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:42.037 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:42.037 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:42.037 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:42.037 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:42.038 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:42.038 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:42.038 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:42.038 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:42.038 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:42.039 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:42.039 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:42.039 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:42.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:42.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:42.040 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:42.040 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:42.040 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:42.040 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.103309_245.1", 2021-09-17T02:29:42.041 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:42.041 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:42.041 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:42.041 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.103309Z", 2021-09-17T02:29:42.042 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:42.042 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:42.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:42.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:42.043 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:42.043 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:42.043 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:42.043 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:42.044 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:42.044 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:42.044 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:42.044 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:42.045 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:42.045 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:42.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:42.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:42.045 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:42.046 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:42.046 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:42.046 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:42.047 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:42.047 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:42.047 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:42.047 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:42.048 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:42.048 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:42.048 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:42.049 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:42.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:42.049 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:42.050 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:42.050 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:42.050 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:42.050 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:42.051 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.103309_245.1 2021-09-17T02:29:47.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:47.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:47.054 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:29:47.125 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:29:47.125 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:47.178 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:47.178 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:47.179 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:47.179 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:47.179 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:47.179 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:47.180 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:47.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:47.180 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:47.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:47.181 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:47.181 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:47.181 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:47.181 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:47.182 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:47.182 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:47.182 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:47.182 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:47.183 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:47.183 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:47.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:47.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:47.184 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:47.184 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:47.184 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:47.184 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.266133_251.1", 2021-09-17T02:29:47.185 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:47.185 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:47.185 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:47.185 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.266133Z", 2021-09-17T02:29:47.186 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:47.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:47.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:47.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:47.187 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:47.187 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:47.188 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:47.188 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:47.188 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:47.188 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:47.189 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:47.189 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:47.189 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:47.189 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:47.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:47.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:47.190 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:47.190 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:47.190 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:47.191 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:47.191 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:47.191 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:47.191 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:47.191 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:47.192 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:47.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:47.192 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:47.192 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:47.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:47.193 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:47.193 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:47.193 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:47.193 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:47.194 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:47.195 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.266133_251.1 2021-09-17T02:29:52.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:52.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:52.197 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:29:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:29:52.386 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:52.447 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.266133_251.1", 2021-09-17T02:29:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.266133Z", 2021-09-17T02:29:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:52.460 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:52.460 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:52.460 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:52.460 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:52.462 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:52.462 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:52.462 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:52.463 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.266133_251.1 2021-09-17T02:29:57.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:57.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:29:57.466 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:29:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:29:57.516 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:29:57.576 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:29:57.576 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:29:57.577 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:29:57.577 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:29:57.577 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:29:57.577 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:29:57.578 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:57.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:57.578 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:29:57.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:57.579 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:29:57.579 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:57.579 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:57.580 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:29:57.580 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:57.581 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:29:57.581 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:57.581 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:29:57.581 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:57.582 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:57.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:57.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:57.582 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:29:57.582 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:57.583 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:57.583 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.268651_252.1", 2021-09-17T02:29:57.583 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:57.583 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:57.583 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:57.584 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.268651Z", 2021-09-17T02:29:57.584 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:57.584 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:57.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:57.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:57.585 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:29:57.585 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:57.585 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:57.585 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:57.586 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:57.586 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:57.586 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:57.586 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:57.587 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:57.587 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:57.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:57.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:29:57.587 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:29:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:29:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:29:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:29:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:29:57.588 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:29:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:29:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:29:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:29:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:57.589 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:29:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:29:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:29:57.590 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:29:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:29:57.591 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:29:57.591 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:29:57.592 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.268651_252.1 2021-09-17T02:30:02.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:02.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:02.595 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:02.647 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:02.647 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:02.708 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:02.708 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:02.709 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:02.709 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:02.709 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:02.709 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:02.709 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:02.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:02.710 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:02.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:02.710 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:02.711 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:02.711 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:02.711 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:02.711 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:02.711 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:02.712 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:02.712 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:02.712 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:02.712 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:02.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:02.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:02.713 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:02.713 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:02.713 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:02.713 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.272193_254.1", 2021-09-17T02:30:02.714 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:02.714 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:02.714 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:02.714 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.272193Z", 2021-09-17T02:30:02.715 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:02.715 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:02.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:02.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:02.716 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:02.716 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:02.716 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:02.716 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:02.716 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:02.717 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:02.717 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:02.717 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:02.717 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:02.718 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:02.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:02.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:02.718 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:02.718 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:02.719 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:02.719 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:02.719 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:02.719 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:02.720 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:02.720 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:02.720 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:02.720 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:02.720 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:02.721 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:02.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:02.721 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:02.721 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:02.721 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:02.722 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:02.722 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:02.723 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.272193_254.1 2021-09-17T02:30:07.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:07.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:07.726 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:07.772 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:07.773 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:07.828 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:07.829 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:07.829 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:07.829 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:07.829 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:07.830 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:07.830 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:07.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:07.830 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:07.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:07.831 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:07.831 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:07.831 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:07.831 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:07.831 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:07.832 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:07.832 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:07.833 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:07.833 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:07.833 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:07.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:07.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:07.834 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:07.834 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:07.834 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:07.834 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:30:07.835 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:07.835 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:07.835 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:07.835 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:30:07.835 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:07.836 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:07.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:07.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:07.836 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:07.836 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:07.837 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:07.837 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:07.837 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:07.837 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:07.837 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:07.838 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:07.838 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:07.838 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:07.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:07.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:07.839 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:07.839 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:07.839 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:07.839 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:07.840 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:07.840 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:07.840 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:07.840 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:07.840 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:07.841 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:07.841 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:07.841 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:07.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:07.841 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:07.842 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:07.842 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:07.842 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:07.842 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:07.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:30:12.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:12.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:12.847 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:12.895 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:12.895 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:12.954 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:12.954 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:12.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:12.956 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:12.957 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:12.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.161433_248.1", 2021-09-17T02:30:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.161433Z", 2021-09-17T02:30:12.960 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:12.961 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:12.962 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:12.963 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:12.964 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:12.965 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:12.966 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:12.966 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:12.966 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:12.966 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:12.966 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:12.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:12.967 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:12.967 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:12.967 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:12.968 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:12.968 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:12.969 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.161433_248.1 2021-09-17T02:30:17.970 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:17.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:17.971 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:18.023 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:18.023 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:18.085 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:18.085 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:18.085 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:18.085 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:18.086 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:18.086 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:18.086 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:18.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:18.086 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:18.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:18.087 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:18.087 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:18.087 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:18.088 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:18.088 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:18.088 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:18.088 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:18.088 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:18.089 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:18.089 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:18.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:18.089 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:18.089 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:18.090 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:18.090 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:18.090 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.209532_249.1", 2021-09-17T02:30:18.090 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:18.090 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:18.091 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:18.091 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.209532Z", 2021-09-17T02:30:18.091 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:18.091 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:18.091 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:18.092 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:18.092 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:18.092 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:18.092 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:18.092 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:18.093 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:18.093 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:18.093 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:18.093 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:18.093 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:18.094 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:18.094 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:18.094 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:18.094 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:18.095 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:18.095 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:18.095 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:18.095 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:18.095 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:18.096 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:18.096 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:18.096 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:18.096 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:18.096 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:18.097 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:18.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:18.097 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:18.097 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:18.097 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:18.098 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:18.098 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:18.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.209532_249.1 2021-09-17T02:30:23.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:23.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:23.102 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:23.151 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:23.152 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:23.213 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:23.213 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:23.214 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:23.214 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:23.214 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:23.214 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:23.215 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:23.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:23.215 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:23.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:23.216 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:23.216 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:23.216 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:23.216 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:23.216 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:23.217 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:23.217 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:23.217 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:23.217 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:23.218 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:23.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:23.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:23.218 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:23.218 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:23.219 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:23.219 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.288419_257.1", 2021-09-17T02:30:23.219 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:23.219 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:23.220 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:23.220 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.288419Z", 2021-09-17T02:30:23.220 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:23.220 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:23.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:23.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:23.221 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:23.221 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:23.221 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:23.222 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:23.222 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:23.222 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:23.222 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:23.222 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:23.223 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:23.223 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:23.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:23.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:23.224 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:23.224 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:23.224 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:23.224 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:23.224 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:23.225 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:23.225 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:23.225 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:23.225 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:23.226 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:23.226 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:23.226 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:23.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:23.227 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:23.227 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:23.227 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:23.227 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:23.227 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:23.229 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.288419_257.1 2021-09-17T02:30:28.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:28.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:28.231 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:28.296 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:28.296 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:28.358 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:28.359 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:28.359 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:28.359 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:28.359 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:28.360 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:28.360 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:28.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:28.360 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:28.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:28.361 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:28.361 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:28.361 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:28.361 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:28.362 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:28.362 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:28.362 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:28.362 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:28.363 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:28.363 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:28.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:28.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:28.364 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:28.364 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:28.364 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:28.364 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.103309_245.1", 2021-09-17T02:30:28.365 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:28.365 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:28.365 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:28.365 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.103309Z", 2021-09-17T02:30:28.366 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:28.366 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:28.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:28.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:28.366 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:28.367 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:28.367 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:28.367 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:28.367 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:28.368 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:28.368 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:28.368 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:28.368 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:28.369 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:28.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:28.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:28.369 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:28.370 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:28.370 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:28.370 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:28.370 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:28.371 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:28.371 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:28.371 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:28.371 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:28.371 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:28.372 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:28.372 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:28.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:28.372 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:28.373 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:28.373 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:28.373 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:28.373 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:28.374 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.103309_245.1 2021-09-17T02:30:33.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:33.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:33.378 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:33.428 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:33.429 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:33.504 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:33.505 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:33.505 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:33.505 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:33.505 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:33.506 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:33.506 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:33.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:33.506 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:33.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:33.507 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:33.507 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:33.507 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:33.507 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:33.508 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:33.508 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:33.508 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:33.508 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:33.509 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:33.509 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:33.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:33.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:33.510 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:33.510 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:33.510 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:33.510 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.105759_246.1", 2021-09-17T02:30:33.510 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:33.511 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:33.511 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:33.511 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.105759Z", 2021-09-17T02:30:33.511 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:33.512 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:33.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:33.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:33.512 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:33.513 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:33.513 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:33.513 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:33.513 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:33.514 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:33.514 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:33.514 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:33.514 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:33.514 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:33.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:33.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:33.515 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:33.515 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:33.516 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:33.516 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:33.516 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:33.516 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:33.516 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:33.517 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:33.517 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:33.517 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:33.517 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:33.518 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:33.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:33.518 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:33.518 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:33.518 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:33.519 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:33.519 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:33.520 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.105759_246.1 2021-09-17T02:30:38.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:38.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:38.522 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:38.620 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:38.620 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:38.977 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:38.977 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:38.977 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:38.978 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:38.978 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:38.978 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:38.978 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:38.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:38.979 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:38.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:38.979 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:38.979 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:38.980 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:38.980 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:38.980 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:38.980 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:38.980 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:38.981 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:38.981 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:38.981 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:38.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:38.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:38.982 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:38.982 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:38.982 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:38.983 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.157882_247.1", 2021-09-17T02:30:38.983 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:38.983 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:38.983 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:38.984 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.157882Z", 2021-09-17T02:30:38.984 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:38.984 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:38.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:38.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:38.985 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:38.985 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:38.985 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:38.985 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:38.985 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:38.986 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:38.986 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:38.986 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:38.986 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:38.986 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:38.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:38.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:38.987 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:38.987 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:38.988 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:38.988 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:38.988 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:38.988 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:38.988 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:38.989 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:38.989 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:38.989 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:38.989 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:38.989 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:38.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:38.990 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:38.990 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:38.990 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:38.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:38.991 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:38.991 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.157882_247.1 2021-09-17T02:30:43.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:43.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:43.994 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:44.065 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:44.066 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:44.164 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:44.165 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:44.165 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:44.165 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:44.166 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:44.166 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:44.166 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:44.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:44.166 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:44.167 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:44.167 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:44.167 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:44.167 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:44.168 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:44.168 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:44.168 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:44.168 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:44.169 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:44.169 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:44.169 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:44.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:44.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:44.170 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:44.170 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:44.170 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:44.171 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.086132_241.1", 2021-09-17T02:30:44.171 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:44.171 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:44.171 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:44.172 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.086132Z", 2021-09-17T02:30:44.172 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:44.172 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:44.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:44.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:44.173 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:44.173 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:44.173 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:44.174 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:44.174 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:44.174 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:44.174 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:44.175 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:44.175 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:44.175 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:44.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:44.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:44.176 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:44.176 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:44.176 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:44.176 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:44.177 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:44.177 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:44.177 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:44.177 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:44.178 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:44.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:44.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:44.178 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:44.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:44.179 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:44.179 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:44.179 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:44.180 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:44.180 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:44.181 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.086132_241.1 2021-09-17T02:30:49.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:49.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:49.184 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:49.263 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:49.264 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:49.558 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:49.559 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:49.559 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:49.559 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:49.559 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:49.560 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:49.560 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:49.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:49.560 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:49.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:49.561 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:49.561 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:49.562 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:49.562 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:49.562 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:49.562 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:49.563 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:49.563 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:49.563 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:49.563 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:49.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:49.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:49.564 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:49.564 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:49.564 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:49.565 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:30:49.565 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:49.565 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:49.565 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:49.565 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:30:49.566 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:49.566 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:49.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:49.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:49.566 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:49.567 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:49.567 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:49.567 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:49.568 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:49.568 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:49.568 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:49.568 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:49.569 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:49.569 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:49.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:49.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:49.570 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:49.570 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:49.571 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:49.571 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:49.571 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:49.572 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:49.572 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:49.572 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:49.572 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:49.572 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:49.573 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:49.573 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:49.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:49.573 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:49.574 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:49.574 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:49.574 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:49.574 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:49.575 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.088776_242.1 2021-09-17T02:30:54.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:54.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:54.578 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:54.663 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:54.664 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:54.784 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:54.784 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:54.784 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:54.785 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:54.785 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:54.785 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:54.785 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:54.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:54.786 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:54.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:54.786 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:54.786 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:54.787 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:54.787 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:54.787 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:54.787 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:54.788 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:54.788 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:54.788 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:54.788 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:54.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:54.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:54.789 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:54.789 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:54.789 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:54.790 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.161433_248.1", 2021-09-17T02:30:54.790 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:54.790 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:54.790 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:54.791 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.161433Z", 2021-09-17T02:30:54.791 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:54.791 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:54.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:54.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:54.792 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:54.792 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:54.792 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:54.792 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:54.793 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:54.793 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:54.793 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:54.793 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:54.793 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:54.794 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:54.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:54.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:54.794 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:54.795 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:54.795 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:54.795 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:54.795 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:54.795 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:54.796 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:54.796 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:54.796 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:54.796 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:54.797 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:54.797 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:54.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:54.797 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:54.798 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:54.798 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:54.798 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:54.798 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:54.800 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.161433_248.1 2021-09-17T02:30:59.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:59.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:30:59.803 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:30:59.848 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:30:59.849 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:30:59.901 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:30:59.902 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:30:59.902 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:30:59.902 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:30:59.902 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:30:59.903 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:30:59.903 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:59.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:59.903 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:30:59.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:59.904 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:30:59.904 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:59.904 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:59.904 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:30:59.905 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:59.905 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:30:59.905 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:59.905 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:30:59.905 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:59.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:59.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:59.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:59.906 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:30:59.906 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:59.907 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:59.907 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:30:59.907 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:59.907 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:59.908 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:59.908 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:30:59.908 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:59.908 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:59.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:59.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:59.909 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:30:59.909 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:59.909 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:59.909 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:59.910 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:59.910 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:59.910 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:59.911 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:59.911 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:59.911 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:59.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:59.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:30:59.912 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:30:59.912 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:30:59.912 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:30:59.913 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:30:59.913 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:30:59.914 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:30:59.914 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:30:59.915 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:30:59.915 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:30:59.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:59.916 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:59.916 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:30:59.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:30:59.916 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:30:59.917 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:30:59.917 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:30:59.917 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:30:59.918 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:30:59.920 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:31:04.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:31:04.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:31:04.923 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:31:04.967 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:31:04.968 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:31:05.020 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:31:05.021 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:31:05.021 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:31:05.021 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:31:05.021 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:31:05.022 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:31:05.022 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:05.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:05.022 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:31:05.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:05.023 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:31:05.023 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:05.023 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:05.023 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:31:05.024 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:05.024 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:31:05.024 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:05.024 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:31:05.025 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:05.025 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:05.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:05.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:05.025 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:31:05.026 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:05.026 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:05.026 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.091993_244.1", 2021-09-17T02:31:05.026 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:05.026 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:31:05.027 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:05.027 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.091993Z", 2021-09-17T02:31:05.027 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:05.027 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:05.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:05.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:05.028 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:31:05.028 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:05.029 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:05.029 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:31:05.029 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:05.029 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:31:05.030 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:05.030 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:31:05.030 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:05.030 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:05.031 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:05.031 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:05.031 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:31:05.031 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:05.031 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:05.032 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:31:05.032 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:05.032 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:31:05.032 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:05.033 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:31:05.033 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:05.033 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:05.034 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:05.034 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:31:05.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:05.034 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:31:05.035 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:31:05.035 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:31:05.035 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:05.035 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:31:05.037 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.091993_244.1 2021-09-17T02:31:10.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:31:10.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:31:10.040 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:31:10.085 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:31:10.085 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:31:10.138 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:31:10.139 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:31:10.139 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:31:10.139 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:31:10.140 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:31:10.140 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:31:10.140 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:10.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:10.140 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:31:10.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:10.141 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:31:10.141 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:10.141 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:10.142 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:31:10.142 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:10.142 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:31:10.142 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:10.143 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:31:10.143 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:10.143 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:10.143 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:10.144 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:10.144 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:31:10.144 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:10.144 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:10.144 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.269640_253.1", 2021-09-17T02:31:10.145 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:10.145 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:31:10.145 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:10.145 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.269640Z", 2021-09-17T02:31:10.145 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:10.146 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:10.146 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:10.146 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:10.146 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:31:10.147 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:10.147 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:10.147 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:31:10.147 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:10.147 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:31:10.148 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:10.148 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:31:10.149 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:10.149 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:10.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:10.149 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:10.149 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:31:10.150 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:10.150 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:10.150 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:31:10.150 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:10.151 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:31:10.151 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:10.151 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:31:10.151 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:10.152 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:10.152 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:10.152 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:31:10.152 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:10.152 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:31:10.153 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:31:10.153 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:31:10.153 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:10.153 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:31:10.155 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.269640_253.1 2021-09-17T02:31:15.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:31:15.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:31:15.157 DEBUG:teuthology.orchestra.run.smithi045:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:31:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:31:15.203 INFO:teuthology.orchestra.run.smithi045.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:31:15.258 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2021-09-17T02:31:15.258 INFO:teuthology.orchestra.run.smithi045.stdout: "sync_status": { 2021-09-17T02:31:15.258 INFO:teuthology.orchestra.run.smithi045.stdout: "info": { 2021-09-17T02:31:15.258 INFO:teuthology.orchestra.run.smithi045.stdout: "status": "sync", 2021-09-17T02:31:15.259 INFO:teuthology.orchestra.run.smithi045.stdout: "num_shards": 4, 2021-09-17T02:31:15.259 INFO:teuthology.orchestra.run.smithi045.stdout: "period": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:31:15.259 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:15.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:15.260 INFO:teuthology.orchestra.run.smithi045.stdout: "markers": [ 2021-09-17T02:31:15.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:15.260 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 0, 2021-09-17T02:31:15.260 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:15.261 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:15.261 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844050.260780_112.1", 2021-09-17T02:31:15.261 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:15.261 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 1, 2021-09-17T02:31:15.261 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:15.262 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:00:50.260780Z", 2021-09-17T02:31:15.262 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:15.262 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:15.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 1, 2021-09-17T02:31:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "1_1631844259.088776_242.1", 2021-09-17T02:31:15.264 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:15.264 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:31:15.264 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:15.264 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "2021-09-17T02:04:19.088776Z", 2021-09-17T02:31:15.265 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:15.265 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:15.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:15.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:15.266 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 2, 2021-09-17T02:31:15.266 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:15.266 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:15.266 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:31:15.267 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:15.267 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:31:15.267 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:15.267 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:31:15.267 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:15.268 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:15.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:15.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2021-09-17T02:31:15.268 INFO:teuthology.orchestra.run.smithi045.stdout: "key": 3, 2021-09-17T02:31:15.269 INFO:teuthology.orchestra.run.smithi045.stdout: "val": { 2021-09-17T02:31:15.269 INFO:teuthology.orchestra.run.smithi045.stdout: "state": 1, 2021-09-17T02:31:15.269 INFO:teuthology.orchestra.run.smithi045.stdout: "marker": "", 2021-09-17T02:31:15.269 INFO:teuthology.orchestra.run.smithi045.stdout: "next_step_marker": "", 2021-09-17T02:31:15.269 INFO:teuthology.orchestra.run.smithi045.stdout: "total_entries": 0, 2021-09-17T02:31:15.270 INFO:teuthology.orchestra.run.smithi045.stdout: "pos": 0, 2021-09-17T02:31:15.270 INFO:teuthology.orchestra.run.smithi045.stdout: "timestamp": "0.000000", 2021-09-17T02:31:15.270 INFO:teuthology.orchestra.run.smithi045.stdout: "realm_epoch": 2 2021-09-17T02:31:15.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:15.271 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:15.271 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2021-09-17T02:31:15.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2021-09-17T02:31:15.271 INFO:teuthology.orchestra.run.smithi045.stdout: "full_sync": { 2021-09-17T02:31:15.271 INFO:teuthology.orchestra.run.smithi045.stdout: "total": 1, 2021-09-17T02:31:15.272 INFO:teuthology.orchestra.run.smithi045.stdout: "complete": 1 2021-09-17T02:31:15.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2021-09-17T02:31:15.272 INFO:teuthology.orchestra.run.smithi045.stdout:} 2021-09-17T02:31:15.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631845259.851939_401.1 target=1_1631844259.088776_242.1 2021-09-17T02:31:20.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2021-09-17T02:31:20.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.278 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T02:31:20.330 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:31:20.331 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:31:20.361 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:31:20.362 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "c8a1fb6b-032d-4d6a-b41a-82ba11ef9634", 2021-09-17T02:31:20.362 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "remove_zg", 2021-09-17T02:31:20.362 INFO:teuthology.orchestra.run.smithi013.stdout: "api_name": "remove_zg", 2021-09-17T02:31:20.362 INFO:teuthology.orchestra.run.smithi013.stdout: "is_master": "false", 2021-09-17T02:31:20.362 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [], 2021-09-17T02:31:20.363 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames": [], 2021-09-17T02:31:20.363 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames_s3website": [], 2021-09-17T02:31:20.363 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zone": "", 2021-09-17T02:31:20.363 INFO:teuthology.orchestra.run.smithi013.stdout: "zones": [], 2021-09-17T02:31:20.363 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_targets": [], 2021-09-17T02:31:20.364 INFO:teuthology.orchestra.run.smithi013.stdout: "default_placement": "", 2021-09-17T02:31:20.364 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T02:31:20.364 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_policy": { 2021-09-17T02:31:20.364 INFO:teuthology.orchestra.run.smithi013.stdout: "groups": [] 2021-09-17T02:31:20.364 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.365 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:31:20.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.367 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T02:31:20.410 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:31:20.410 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:31:20.438 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:31:20.439 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "1d1edbe7-3f97-4f02-9486-01a2cdccd387", 2021-09-17T02:31:20.439 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "remove", 2021-09-17T02:31:20.439 INFO:teuthology.orchestra.run.smithi013.stdout: "domain_root": "remove.rgw.meta:root", 2021-09-17T02:31:20.439 INFO:teuthology.orchestra.run.smithi013.stdout: "control_pool": "remove.rgw.control", 2021-09-17T02:31:20.440 INFO:teuthology.orchestra.run.smithi013.stdout: "gc_pool": "remove.rgw.log:gc", 2021-09-17T02:31:20.440 INFO:teuthology.orchestra.run.smithi013.stdout: "lc_pool": "remove.rgw.log:lc", 2021-09-17T02:31:20.440 INFO:teuthology.orchestra.run.smithi013.stdout: "log_pool": "remove.rgw.log", 2021-09-17T02:31:20.440 INFO:teuthology.orchestra.run.smithi013.stdout: "intent_log_pool": "remove.rgw.log:intent", 2021-09-17T02:31:20.440 INFO:teuthology.orchestra.run.smithi013.stdout: "usage_log_pool": "remove.rgw.log:usage", 2021-09-17T02:31:20.441 INFO:teuthology.orchestra.run.smithi013.stdout: "roles_pool": "remove.rgw.meta:roles", 2021-09-17T02:31:20.441 INFO:teuthology.orchestra.run.smithi013.stdout: "reshard_pool": "remove.rgw.log:reshard", 2021-09-17T02:31:20.441 INFO:teuthology.orchestra.run.smithi013.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2021-09-17T02:31:20.441 INFO:teuthology.orchestra.run.smithi013.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2021-09-17T02:31:20.442 INFO:teuthology.orchestra.run.smithi013.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2021-09-17T02:31:20.442 INFO:teuthology.orchestra.run.smithi013.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2021-09-17T02:31:20.442 INFO:teuthology.orchestra.run.smithi013.stdout: "otp_pool": "remove.rgw.otp", 2021-09-17T02:31:20.443 INFO:teuthology.orchestra.run.smithi013.stdout: "system_key": { 2021-09-17T02:31:20.443 INFO:teuthology.orchestra.run.smithi013.stdout: "access_key": "", 2021-09-17T02:31:20.443 INFO:teuthology.orchestra.run.smithi013.stdout: "secret_key": "" 2021-09-17T02:31:20.444 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.444 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_pools": [ 2021-09-17T02:31:20.444 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.444 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "default-placement", 2021-09-17T02:31:20.445 INFO:teuthology.orchestra.run.smithi013.stdout: "val": { 2021-09-17T02:31:20.445 INFO:teuthology.orchestra.run.smithi013.stdout: "index_pool": "remove.rgw.buckets.index", 2021-09-17T02:31:20.445 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": { 2021-09-17T02:31:20.445 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD": { 2021-09-17T02:31:20.445 INFO:teuthology.orchestra.run.smithi013.stdout: "data_pool": "remove.rgw.buckets.data" 2021-09-17T02:31:20.446 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.446 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.446 INFO:teuthology.orchestra.run.smithi013.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2021-09-17T02:31:20.446 INFO:teuthology.orchestra.run.smithi013.stdout: "index_type": 0 2021-09-17T02:31:20.447 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.447 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.447 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.447 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T02:31:20.447 INFO:teuthology.orchestra.run.smithi013.stdout: "notif_pool": "remove.rgw.log:notif" 2021-09-17T02:31:20.448 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:31:20.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.450 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T02:31:20.535 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:31:20.535 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:31:20.594 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:31:20.594 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:31:20.595 INFO:teuthology.orchestra.run.smithi013.stdout: "epoch": 6, 2021-09-17T02:31:20.595 INFO:teuthology.orchestra.run.smithi013.stdout: "predecessor_uuid": "d9cd66ce-d16c-4226-9f7c-60489089c5d1", 2021-09-17T02:31:20.595 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": [], 2021-09-17T02:31:20.595 INFO:teuthology.orchestra.run.smithi013.stdout: "period_map": { 2021-09-17T02:31:20.595 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:31:20.596 INFO:teuthology.orchestra.run.smithi013.stdout: "zonegroups": [ 2021-09-17T02:31:20.596 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.596 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "c8a1fb6b-032d-4d6a-b41a-82ba11ef9634", 2021-09-17T02:31:20.596 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "remove_zg", 2021-09-17T02:31:20.597 INFO:teuthology.orchestra.run.smithi013.stdout: "api_name": "remove_zg", 2021-09-17T02:31:20.597 INFO:teuthology.orchestra.run.smithi013.stdout: "is_master": "false", 2021-09-17T02:31:20.597 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [], 2021-09-17T02:31:20.597 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames": [], 2021-09-17T02:31:20.598 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames_s3website": [], 2021-09-17T02:31:20.598 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zone": "1d1edbe7-3f97-4f02-9486-01a2cdccd387", 2021-09-17T02:31:20.598 INFO:teuthology.orchestra.run.smithi013.stdout: "zones": [ 2021-09-17T02:31:20.598 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.598 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "1d1edbe7-3f97-4f02-9486-01a2cdccd387", 2021-09-17T02:31:20.599 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "remove", 2021-09-17T02:31:20.599 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [], 2021-09-17T02:31:20.599 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T02:31:20.599 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "false", 2021-09-17T02:31:20.600 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:31:20.600 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T02:31:20.600 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "", 2021-09-17T02:31:20.600 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "true", 2021-09-17T02:31:20.600 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [], 2021-09-17T02:31:20.601 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T02:31:20.601 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.601 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.601 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_targets": [ 2021-09-17T02:31:20.601 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.602 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "default-placement", 2021-09-17T02:31:20.602 INFO:teuthology.orchestra.run.smithi013.stdout: "tags": [], 2021-09-17T02:31:20.602 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": [ 2021-09-17T02:31:20.602 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD" 2021-09-17T02:31:20.603 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:31:20.603 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.603 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.603 INFO:teuthology.orchestra.run.smithi013.stdout: "default_placement": "default-placement", 2021-09-17T02:31:20.603 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T02:31:20.604 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_policy": { 2021-09-17T02:31:20.604 INFO:teuthology.orchestra.run.smithi013.stdout: "groups": [] 2021-09-17T02:31:20.604 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.604 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.605 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.605 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "test-zonegroup", 2021-09-17T02:31:20.605 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zonegroup", 2021-09-17T02:31:20.605 INFO:teuthology.orchestra.run.smithi013.stdout: "api_name": "", 2021-09-17T02:31:20.605 INFO:teuthology.orchestra.run.smithi013.stdout: "is_master": "true", 2021-09-17T02:31:20.606 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T02:31:20.606 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T02:31:20.606 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.606 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames": [], 2021-09-17T02:31:20.607 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames_s3website": [], 2021-09-17T02:31:20.607 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:31:20.607 INFO:teuthology.orchestra.run.smithi013.stdout: "zones": [ 2021-09-17T02:31:20.607 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.607 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T02:31:20.608 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone3", 2021-09-17T02:31:20.608 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T02:31:20.608 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8001" 2021-09-17T02:31:20.608 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.608 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T02:31:20.609 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "true", 2021-09-17T02:31:20.609 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:31:20.609 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T02:31:20.609 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "", 2021-09-17T02:31:20.610 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "true", 2021-09-17T02:31:20.610 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [], 2021-09-17T02:31:20.610 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T02:31:20.610 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.610 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.611 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T02:31:20.611 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone2", 2021-09-17T02:31:20.611 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T02:31:20.611 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi045.front.sepia.ceph.com:8000" 2021-09-17T02:31:20.612 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.612 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T02:31:20.612 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "true", 2021-09-17T02:31:20.612 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:31:20.612 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T02:31:20.613 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "", 2021-09-17T02:31:20.613 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "false", 2021-09-17T02:31:20.613 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [], 2021-09-17T02:31:20.613 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T02:31:20.613 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.614 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.614 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:31:20.614 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone1", 2021-09-17T02:31:20.614 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T02:31:20.615 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T02:31:20.615 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.615 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T02:31:20.615 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "true", 2021-09-17T02:31:20.615 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:31:20.616 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T02:31:20.616 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "", 2021-09-17T02:31:20.617 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "true", 2021-09-17T02:31:20.617 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [], 2021-09-17T02:31:20.618 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T02:31:20.618 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.618 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.618 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "e8223d60-810a-44ad-a8ee-67d9caa7bcb0", 2021-09-17T02:31:20.618 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone4", 2021-09-17T02:31:20.619 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T02:31:20.619 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi045.front.sepia.ceph.com:8001" 2021-09-17T02:31:20.619 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.619 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T02:31:20.620 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "true", 2021-09-17T02:31:20.620 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:31:20.620 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T02:31:20.620 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "pubsub", 2021-09-17T02:31:20.620 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "false", 2021-09-17T02:31:20.621 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [ 2021-09-17T02:31:20.621 INFO:teuthology.orchestra.run.smithi013.stdout: "test-zone1" 2021-09-17T02:31:20.621 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.621 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T02:31:20.621 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.622 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.622 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_targets": [ 2021-09-17T02:31:20.622 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.622 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "default-placement", 2021-09-17T02:31:20.623 INFO:teuthology.orchestra.run.smithi013.stdout: "tags": [], 2021-09-17T02:31:20.623 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": [ 2021-09-17T02:31:20.623 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD" 2021-09-17T02:31:20.623 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:31:20.623 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.624 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.624 INFO:teuthology.orchestra.run.smithi013.stdout: "default_placement": "default-placement", 2021-09-17T02:31:20.624 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T02:31:20.624 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_policy": { 2021-09-17T02:31:20.625 INFO:teuthology.orchestra.run.smithi013.stdout: "groups": [] 2021-09-17T02:31:20.625 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.625 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.625 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:20.626 INFO:teuthology.orchestra.run.smithi013.stdout: "short_zone_ids": [ 2021-09-17T02:31:20.626 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.626 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "1d1edbe7-3f97-4f02-9486-01a2cdccd387", 2021-09-17T02:31:20.626 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 951212290 2021-09-17T02:31:20.626 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.627 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.627 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T02:31:20.627 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 4005434042 2021-09-17T02:31:20.627 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.628 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.628 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T02:31:20.628 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 4091795418 2021-09-17T02:31:20.628 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.628 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.629 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:31:20.629 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 803492335 2021-09-17T02:31:20.629 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.629 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:20.630 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "e8223d60-810a-44ad-a8ee-67d9caa7bcb0", 2021-09-17T02:31:20.630 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 3317638424 2021-09-17T02:31:20.630 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.630 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:31:20.631 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.631 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T02:31:20.631 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:31:20.631 INFO:teuthology.orchestra.run.smithi013.stdout: "period_config": { 2021-09-17T02:31:20.631 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_quota": { 2021-09-17T02:31:20.632 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T02:31:20.632 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T02:31:20.632 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T02:31:20.632 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T02:31:20.632 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T02:31:20.633 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.633 INFO:teuthology.orchestra.run.smithi013.stdout: "user_quota": { 2021-09-17T02:31:20.633 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T02:31:20.633 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T02:31:20.634 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T02:31:20.634 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T02:31:20.634 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T02:31:20.634 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:20.634 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:20.635 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T02:31:20.635 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_name": "test-realm", 2021-09-17T02:31:20.635 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:31:20.635 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:31:20.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.637 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T02:31:20.710 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:31:20.710 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:31:20.750 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:31:20.748+0000 7f44cca95380 0 zone id 1d1edbe7-3f97-4f02-9486-01a2cdccd387 is not a part of zonegroup test-zonegroup 2021-09-17T02:31:20.750 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:31:20.749+0000 7f44cca95380 0 zone id 1d1edbe7-3f97-4f02-9486-01a2cdccd387 is not a part of zonegroup default 2021-09-17T02:31:20.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.764 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T02:31:20.826 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:31:20.827 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:31:20.931 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:31:20.929+0000 7fbb63579380 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-09-17T02:31:20.940 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-09-17T02:31:20.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:20.941 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T02:31:21.004 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:31:21.005 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:31:21.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:21.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T02:31:21.087 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T02:31:21.154 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:31:21.154 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:31:21.234 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2021-09-17T02:31:21.234 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:31:21.235 INFO:teuthology.orchestra.run.smithi013.stdout: "epoch": 7, 2021-09-17T02:31:21.235 INFO:teuthology.orchestra.run.smithi013.stdout: "predecessor_uuid": "d9cd66ce-d16c-4226-9f7c-60489089c5d1", 2021-09-17T02:31:21.236 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_status": [], 2021-09-17T02:31:21.236 INFO:teuthology.orchestra.run.smithi013.stdout: "period_map": { 2021-09-17T02:31:21.236 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "3c964c05-2a9c-48af-9858-975bf7b57c1a", 2021-09-17T02:31:21.237 INFO:teuthology.orchestra.run.smithi013.stdout: "zonegroups": [ 2021-09-17T02:31:21.237 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:21.238 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "test-zonegroup", 2021-09-17T02:31:21.238 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zonegroup", 2021-09-17T02:31:21.238 INFO:teuthology.orchestra.run.smithi013.stdout: "api_name": "", 2021-09-17T02:31:21.239 INFO:teuthology.orchestra.run.smithi013.stdout: "is_master": "true", 2021-09-17T02:31:21.239 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T02:31:21.239 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T02:31:21.240 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:21.240 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames": [], 2021-09-17T02:31:21.240 INFO:teuthology.orchestra.run.smithi013.stdout: "hostnames_s3website": [], 2021-09-17T02:31:21.241 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:31:21.241 INFO:teuthology.orchestra.run.smithi013.stdout: "zones": [ 2021-09-17T02:31:21.242 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:21.242 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T02:31:21.243 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone3", 2021-09-17T02:31:21.243 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T02:31:21.243 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8001" 2021-09-17T02:31:21.244 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:21.244 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T02:31:21.245 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "true", 2021-09-17T02:31:21.245 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:31:21.245 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T02:31:21.246 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "", 2021-09-17T02:31:21.246 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "true", 2021-09-17T02:31:21.247 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [], 2021-09-17T02:31:21.247 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T02:31:21.247 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:21.248 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:21.248 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T02:31:21.249 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone2", 2021-09-17T02:31:21.249 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T02:31:21.249 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi045.front.sepia.ceph.com:8000" 2021-09-17T02:31:21.250 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:21.250 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T02:31:21.251 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "true", 2021-09-17T02:31:21.251 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:31:21.252 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T02:31:21.252 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "", 2021-09-17T02:31:21.252 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "false", 2021-09-17T02:31:21.253 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [], 2021-09-17T02:31:21.253 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T02:31:21.254 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:21.254 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:21.254 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:31:21.255 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone1", 2021-09-17T02:31:21.255 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T02:31:21.256 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi013.front.sepia.ceph.com:8000" 2021-09-17T02:31:21.256 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:21.257 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T02:31:21.257 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "true", 2021-09-17T02:31:21.257 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:31:21.258 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T02:31:21.258 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "", 2021-09-17T02:31:21.259 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "true", 2021-09-17T02:31:21.259 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [], 2021-09-17T02:31:21.260 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T02:31:21.260 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:21.260 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:21.261 INFO:teuthology.orchestra.run.smithi013.stdout: "id": "e8223d60-810a-44ad-a8ee-67d9caa7bcb0", 2021-09-17T02:31:21.261 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "test-zone4", 2021-09-17T02:31:21.261 INFO:teuthology.orchestra.run.smithi013.stdout: "endpoints": [ 2021-09-17T02:31:21.262 INFO:teuthology.orchestra.run.smithi013.stdout: "http://smithi045.front.sepia.ceph.com:8001" 2021-09-17T02:31:21.262 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:21.262 INFO:teuthology.orchestra.run.smithi013.stdout: "log_meta": "false", 2021-09-17T02:31:21.263 INFO:teuthology.orchestra.run.smithi013.stdout: "log_data": "true", 2021-09-17T02:31:21.263 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_index_max_shards": 11, 2021-09-17T02:31:21.263 INFO:teuthology.orchestra.run.smithi013.stdout: "read_only": "false", 2021-09-17T02:31:21.264 INFO:teuthology.orchestra.run.smithi013.stdout: "tier_type": "pubsub", 2021-09-17T02:31:21.264 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from_all": "false", 2021-09-17T02:31:21.264 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_from": [ 2021-09-17T02:31:21.265 INFO:teuthology.orchestra.run.smithi013.stdout: "test-zone1" 2021-09-17T02:31:21.265 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:21.265 INFO:teuthology.orchestra.run.smithi013.stdout: "redirect_zone": "" 2021-09-17T02:31:21.265 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:21.266 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:21.266 INFO:teuthology.orchestra.run.smithi013.stdout: "placement_targets": [ 2021-09-17T02:31:21.266 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:21.266 INFO:teuthology.orchestra.run.smithi013.stdout: "name": "default-placement", 2021-09-17T02:31:21.267 INFO:teuthology.orchestra.run.smithi013.stdout: "tags": [], 2021-09-17T02:31:21.267 INFO:teuthology.orchestra.run.smithi013.stdout: "storage_classes": [ 2021-09-17T02:31:21.267 INFO:teuthology.orchestra.run.smithi013.stdout: "STANDARD" 2021-09-17T02:31:21.267 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:31:21.267 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:21.268 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:21.268 INFO:teuthology.orchestra.run.smithi013.stdout: "default_placement": "default-placement", 2021-09-17T02:31:21.268 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T02:31:21.268 INFO:teuthology.orchestra.run.smithi013.stdout: "sync_policy": { 2021-09-17T02:31:21.268 INFO:teuthology.orchestra.run.smithi013.stdout: "groups": [] 2021-09-17T02:31:21.269 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:21.269 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:21.269 INFO:teuthology.orchestra.run.smithi013.stdout: ], 2021-09-17T02:31:21.269 INFO:teuthology.orchestra.run.smithi013.stdout: "short_zone_ids": [ 2021-09-17T02:31:21.269 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:21.271 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "505278ea-3c6d-4c68-8442-578099047a49", 2021-09-17T02:31:21.271 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 4005434042 2021-09-17T02:31:21.271 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:21.271 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:21.272 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "54e01141-a1e3-4035-9545-e42ec73c8edc", 2021-09-17T02:31:21.272 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 4091795418 2021-09-17T02:31:21.272 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:21.272 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:21.273 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:31:21.273 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 803492335 2021-09-17T02:31:21.273 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:21.273 INFO:teuthology.orchestra.run.smithi013.stdout: { 2021-09-17T02:31:21.273 INFO:teuthology.orchestra.run.smithi013.stdout: "key": "e8223d60-810a-44ad-a8ee-67d9caa7bcb0", 2021-09-17T02:31:21.274 INFO:teuthology.orchestra.run.smithi013.stdout: "val": 3317638424 2021-09-17T02:31:21.274 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:21.274 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2021-09-17T02:31:21.274 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:21.275 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T02:31:21.275 INFO:teuthology.orchestra.run.smithi013.stdout: "master_zone": "95b0d488-6c59-4a37-b3ed-d5fa65f27678", 2021-09-17T02:31:21.275 INFO:teuthology.orchestra.run.smithi013.stdout: "period_config": { 2021-09-17T02:31:21.275 INFO:teuthology.orchestra.run.smithi013.stdout: "bucket_quota": { 2021-09-17T02:31:21.275 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T02:31:21.276 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T02:31:21.276 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T02:31:21.276 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T02:31:21.276 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T02:31:21.276 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:21.277 INFO:teuthology.orchestra.run.smithi013.stdout: "user_quota": { 2021-09-17T02:31:21.277 INFO:teuthology.orchestra.run.smithi013.stdout: "enabled": false, 2021-09-17T02:31:21.278 INFO:teuthology.orchestra.run.smithi013.stdout: "check_on_raw": false, 2021-09-17T02:31:21.278 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size": -1, 2021-09-17T02:31:21.278 INFO:teuthology.orchestra.run.smithi013.stdout: "max_size_kb": 0, 2021-09-17T02:31:21.278 INFO:teuthology.orchestra.run.smithi013.stdout: "max_objects": -1 2021-09-17T02:31:21.278 INFO:teuthology.orchestra.run.smithi013.stdout: } 2021-09-17T02:31:21.279 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2021-09-17T02:31:21.279 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_id": "701b5af2-9aae-4e89-bd12-7ed067bbad31", 2021-09-17T02:31:21.279 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_name": "test-realm", 2021-09-17T02:31:21.279 INFO:teuthology.orchestra.run.smithi013.stdout: "realm_epoch": 2 2021-09-17T02:31:21.279 INFO:teuthology.orchestra.run.smithi013.stdout:} 2021-09-17T02:31:21.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-09-17T02:31:21.281 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-46 2021-09-17T02:31:31.044 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=qdiasb-47 2021-09-17T02:31:31.135 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=qdiasb-48 2021-09-17T02:31:31.195 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-49 2021-09-17T02:31:31.197 INFO:tasks.rgw.c1.client.0.smithi013.stdout:*** Caught signal (Segmentation fault) ** 2021-09-17T02:31:31.197 INFO:tasks.rgw.c1.client.0.smithi013.stdout: in thread 7fb0b4abd700 thread_name:radosgw 2021-09-17T02:31:31.198 INFO:tasks.rgw.c1.client.0.smithi013.stdout: ceph version 17.0.0-7730-g2a911e49 (2a911e49300877ce783c2becf9f484a85fa1c049) quincy (dev) 2021-09-17T02:31:31.198 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb141e2db20] 2021-09-17T02:31:31.198 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7fb144a2a97e] 2021-09-17T02:31:31.198 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7fb144a486e4] 2021-09-17T02:31:31.199 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 4: /lib64/libradosgw.so.2(+0x533e4d) [0x7fb14497be4d] 2021-09-17T02:31:31.199 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 5: /lib64/libradosgw.so.2(+0x535430) [0x7fb14497d430] 2021-09-17T02:31:31.199 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 6: /lib64/libradosgw.so.2(+0x53558c) [0x7fb14497d58c] 2021-09-17T02:31:31.199 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 7: make_fcontext() 2021-09-17T02:31:31.200 INFO:tasks.rgw.c1.client.0.smithi013.stdout:2021-09-17T02:31:31.194+0000 7fb0b4abd700 -1 *** Caught signal (Segmentation fault) ** 2021-09-17T02:31:31.200 INFO:tasks.rgw.c1.client.0.smithi013.stdout: in thread 7fb0b4abd700 thread_name:radosgw 2021-09-17T02:31:31.200 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 2021-09-17T02:31:31.200 INFO:tasks.rgw.c1.client.0.smithi013.stdout: ceph version 17.0.0-7730-g2a911e49 (2a911e49300877ce783c2becf9f484a85fa1c049) quincy (dev) 2021-09-17T02:31:31.200 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb141e2db20] 2021-09-17T02:31:31.201 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7fb144a2a97e] 2021-09-17T02:31:31.201 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7fb144a486e4] 2021-09-17T02:31:31.201 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 4: /lib64/libradosgw.so.2(+0x533e4d) [0x7fb14497be4d] 2021-09-17T02:31:31.201 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 5: /lib64/libradosgw.so.2(+0x535430) [0x7fb14497d430] 2021-09-17T02:31:31.202 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 6: /lib64/libradosgw.so.2(+0x53558c) [0x7fb14497d58c] 2021-09-17T02:31:31.202 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 7: make_fcontext() 2021-09-17T02:31:31.202 INFO:tasks.rgw.c1.client.0.smithi013.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-17T02:31:31.202 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 2021-09-17T02:31:31.214 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 0> 2021-09-17T02:31:31.194+0000 7fb0b4abd700 -1 *** Caught signal (Segmentation fault) ** 2021-09-17T02:31:31.214 INFO:tasks.rgw.c1.client.0.smithi013.stdout: in thread 7fb0b4abd700 thread_name:radosgw 2021-09-17T02:31:31.215 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 2021-09-17T02:31:31.215 INFO:tasks.rgw.c1.client.0.smithi013.stdout: ceph version 17.0.0-7730-g2a911e49 (2a911e49300877ce783c2becf9f484a85fa1c049) quincy (dev) 2021-09-17T02:31:31.215 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb141e2db20] 2021-09-17T02:31:31.215 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7fb144a2a97e] 2021-09-17T02:31:31.216 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7fb144a486e4] 2021-09-17T02:31:31.216 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 4: /lib64/libradosgw.so.2(+0x533e4d) [0x7fb14497be4d] 2021-09-17T02:31:31.216 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 5: /lib64/libradosgw.so.2(+0x535430) [0x7fb14497d430] 2021-09-17T02:31:31.216 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 6: /lib64/libradosgw.so.2(+0x53558c) [0x7fb14497d58c] 2021-09-17T02:31:31.216 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 7: make_fcontext() 2021-09-17T02:31:31.217 INFO:tasks.rgw.c1.client.0.smithi013.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-17T02:31:31.217 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 2021-09-17T02:31:31.229 INFO:tasks.rgw.c1.client.0.smithi013.stdout: -36> 2021-09-17T02:31:31.194+0000 7fb0b4abd700 -1 *** Caught signal (Segmentation fault) ** 2021-09-17T02:31:31.229 INFO:tasks.rgw.c1.client.0.smithi013.stdout: in thread 7fb0b4abd700 thread_name:radosgw 2021-09-17T02:31:31.230 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 2021-09-17T02:31:31.230 INFO:tasks.rgw.c1.client.0.smithi013.stdout: ceph version 17.0.0-7730-g2a911e49 (2a911e49300877ce783c2becf9f484a85fa1c049) quincy (dev) 2021-09-17T02:31:31.230 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb141e2db20] 2021-09-17T02:31:31.230 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7fb144a2a97e] 2021-09-17T02:31:31.231 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7fb144a486e4] 2021-09-17T02:31:31.231 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 4: /lib64/libradosgw.so.2(+0x533e4d) [0x7fb14497be4d] 2021-09-17T02:31:31.231 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 5: /lib64/libradosgw.so.2(+0x535430) [0x7fb14497d430] 2021-09-17T02:31:31.231 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 6: /lib64/libradosgw.so.2(+0x53558c) [0x7fb14497d58c] 2021-09-17T02:31:31.232 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 7: make_fcontext() 2021-09-17T02:31:31.232 INFO:tasks.rgw.c1.client.0.smithi013.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-17T02:31:31.232 INFO:tasks.rgw.c1.client.0.smithi013.stdout: 2021-09-17T02:31:31.360 INFO:tasks.rgw.c1.client.0.smithi013.stderr:daemon-helper: command crashed with signal 11 2021-09-17T02:31:33.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-17T02:31:38.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2021-09-17T02:31:44.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~11s 2021-09-17T02:31:49.961 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~17s 2021-09-17T02:31:55.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~22s 2021-09-17T02:32:01.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~28s 2021-09-17T02:32:06.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~34s 2021-09-17T02:32:12.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~39s 2021-09-17T02:32:17.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~45s 2021-09-17T02:32:23.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~50s 2021-09-17T02:32:29.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~56s 2021-09-17T02:32:34.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~62s 2021-09-17T02:32:40.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~67s 2021-09-17T02:32:43.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... rgw_multi.tests.test_set_bucket_policy ... ERROR 2021-09-17T02:32:43.808 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-50 2021-09-17T02:32:46.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~73s 2021-09-17T02:32:51.624 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~78s 2021-09-17T02:32:57.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~84s 2021-09-17T02:33:02.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~90s 2021-09-17T02:33:08.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~95s 2021-09-17T02:33:14.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~101s 2021-09-17T02:33:19.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~107s 2021-09-17T02:33:25.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~112s 2021-09-17T02:33:30.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~118s 2021-09-17T02:33:35.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ERROR 2021-09-17T02:33:35.123 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-51 2021-09-17T02:33:36.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~123s 2021-09-17T02:33:42.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~129s 2021-09-17T02:33:47.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~135s 2021-09-17T02:33:53.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~140s 2021-09-17T02:33:58.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~146s 2021-09-17T02:34:04.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~151s 2021-09-17T02:34:10.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~157s 2021-09-17T02:34:15.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~163s 2021-09-17T02:34:21.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~168s 2021-09-17T02:34:26.917 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~174s 2021-09-17T02:34:32.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~180s 2021-09-17T02:34:36.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ERROR 2021-09-17T02:34:36.533 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-52 2021-09-17T02:34:38.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~185s 2021-09-17T02:34:44.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~191s 2021-09-17T02:34:49.706 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~197s 2021-09-17T02:34:55.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~202s 2021-09-17T02:35:00.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~208s 2021-09-17T02:35:06.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~213s 2021-09-17T02:35:08.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ERROR 2021-09-17T02:35:08.433 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-53 2021-09-17T02:35:12.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~219s 2021-09-17T02:35:17.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~225s 2021-09-17T02:35:23.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~230s 2021-09-17T02:35:28.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~236s 2021-09-17T02:35:34.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~241s 2021-09-17T02:35:40.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~247s 2021-09-17T02:35:45.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~253s 2021-09-17T02:35:51.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~258s 2021-09-17T02:35:56.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~264s 2021-09-17T02:36:02.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~269s 2021-09-17T02:36:08.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~275s 2021-09-17T02:36:13.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~281s 2021-09-17T02:36:19.390 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~286s 2021-09-17T02:36:23.577 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ERROR 2021-09-17T02:36:23.578 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-54 2021-09-17T02:36:24.997 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~292s 2021-09-17T02:36:30.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~297s 2021-09-17T02:36:36.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~303s 2021-09-17T02:36:36.207 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2021-09-17T02:36:36.811 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2021-09-17T02:36:36.813 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2021-09-17T02:36:36.814 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2021-09-17T02:36:36.814 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2021-09-17T02:36:36.815 INFO:tasks.rgw.c1.client.1:Sent signal 15 2021-09-17T02:36:36.815 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2021-09-17T02:36:36.816 INFO:tasks.ceph.c1.osd.0.smithi013.stderr:2021-09-17T02:36:36.811+0000 7ff37c208700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 31445) UID: 0 2021-09-17T02:36:36.816 INFO:tasks.ceph.c1.osd.0.smithi013.stderr:2021-09-17T02:36:36.811+0000 7ff37c208700 -1 osd.0 61 *** Got signal Terminated *** 2021-09-17T02:36:36.817 INFO:tasks.ceph.c1.osd.0.smithi013.stderr:2021-09-17T02:36:36.811+0000 7ff37c208700 -1 osd.0 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-17T02:36:36.817 INFO:tasks.ceph.c1.osd.1.smithi013.stderr:2021-09-17T02:36:36.813+0000 7f69066b2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 31447) UID: 0 2021-09-17T02:36:36.818 INFO:tasks.ceph.c1.osd.1.smithi013.stderr:2021-09-17T02:36:36.813+0000 7f69066b2700 -1 osd.1 61 *** Got signal Terminated *** 2021-09-17T02:36:36.818 INFO:tasks.ceph.c1.osd.1.smithi013.stderr:2021-09-17T02:36:36.813+0000 7f69066b2700 -1 osd.1 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-17T02:36:36.818 INFO:tasks.ceph.c1.osd.2.smithi013.stderr:2021-09-17T02:36:36.813+0000 7f568a803700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 31450) UID: 0 2021-09-17T02:36:36.819 INFO:tasks.ceph.c1.osd.2.smithi013.stderr:2021-09-17T02:36:36.813+0000 7f568a803700 -1 osd.2 61 *** Got signal Terminated *** 2021-09-17T02:36:36.819 INFO:tasks.ceph.c1.osd.2.smithi013.stderr:2021-09-17T02:36:36.813+0000 7f568a803700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-17T02:36:36.819 INFO:tasks.ceph.c1.mon.a.smithi013.stderr:2021-09-17T02:36:36.814+0000 7fdfb700d700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 31125) UID: 0 2021-09-17T02:36:36.820 INFO:tasks.ceph.c1.mon.a.smithi013.stderr:2021-09-17T02:36:36.814+0000 7fdfb700d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-09-17T02:36:36.820 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:36:36.814+0000 7f1e2d04d700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 39685) UID: 0 2021-09-17T02:36:36.821 INFO:tasks.rgw.c1.client.1.smithi013.stdout:2021-09-17T02:36:36.814+0000 7f1e3deee440 -1 shutting down 2021-09-17T02:36:36.821 INFO:tasks.ceph.c1.mgr.x.smithi013.stderr:2021-09-17T02:36:36.814+0000 7efca4d0e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 31129) UID: 0 2021-09-17T02:36:36.821 INFO:tasks.ceph.c1.mgr.x.smithi013.stderr:2021-09-17T02:36:36.814+0000 7efca4d0e700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2021-09-17T02:37:39.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2021-09-17T02:37:39.853 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=qdiasb-55 2021-09-17T02:38:15.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2021-09-17T02:38:15.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:38:15.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:38:15.282 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:38:15.335 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:38:15.335 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:43:15.338 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:43:15.335+0000 7f68507c9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:48:15.340 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:48:15.335+0000 7f68507c9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:53:15.340 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:53:15.337+0000 7f68507c9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:58:15.341 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T02:58:15.338+0000 7f68507c9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:03:15.341 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:03:15.338+0000 7f68507c9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:08:15.341 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:08:15.339+0000 7f68507c9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:13:15.342 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:13:15.338+0000 7f68507c9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:18:15.343 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:18:15.339+0000 7f68507c9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:23:15.343 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:23:15.341+0000 7f68507c9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:28:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T03:28:15.348 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:28:15.341+0000 7f68507c9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:28:15.348 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T03:28:15.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2021-09-17T03:28:15.351 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-09-17T03:28:15.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T03:28:15.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T03:28:15.358 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T03:28:15.413 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T03:28:15.413 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T03:33:15.417 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:33:15.412+0000 7fe15b77c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:38:15.416 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:38:15.412+0000 7fe15b77c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:43:15.416 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:43:15.414+0000 7fe15b77c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:48:15.417 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:48:15.413+0000 7fe15b77c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:53:15.418 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:53:15.413+0000 7fe15b77c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:58:15.418 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T03:58:15.414+0000 7fe15b77c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:03:15.419 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:03:15.416+0000 7fe15b77c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:08:15.419 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:08:15.416+0000 7fe15b77c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:13:15.420 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:13:15.416+0000 7fe15b77c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:18:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T04:18:15.432 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:18:15.416+0000 7fe15b77c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:18:15.432 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T04:18:15.433 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2021-09-17T04:18:15.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T04:18:15.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T04:18:15.437 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T04:18:15.488 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T04:18:15.489 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T04:23:15.491 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:23:15.488+0000 7f7306cc9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:28:15.491 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:28:15.488+0000 7f7306cc9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:33:15.492 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:33:15.489+0000 7f7306cc9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:38:15.492 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:38:15.489+0000 7f7306cc9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:43:15.493 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:43:15.490+0000 7f7306cc9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:48:15.493 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:48:15.489+0000 7f7306cc9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:53:15.494 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:53:15.490+0000 7f7306cc9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:58:15.495 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T04:58:15.491+0000 7f7306cc9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:03:15.495 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:03:15.492+0000 7f7306cc9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:08:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T05:08:15.500 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:08:15.493+0000 7f7306cc9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:08:15.500 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T05:08:15.502 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2021-09-17T05:08:15.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T05:08:15.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T05:08:15.505 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T05:08:15.559 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T05:08:15.560 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T05:13:15.561 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:13:15.557+0000 7f2a6c693380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:18:15.562 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:18:15.559+0000 7f2a6c693380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:23:15.561 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:23:15.559+0000 7f2a6c693380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:28:15.562 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:28:15.559+0000 7f2a6c693380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:33:15.563 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:33:15.559+0000 7f2a6c693380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:38:15.563 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:38:15.560+0000 7f2a6c693380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:43:15.564 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:43:15.561+0000 7f2a6c693380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:48:15.564 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:48:15.561+0000 7f2a6c693380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:53:15.564 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:53:15.561+0000 7f2a6c693380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:58:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T05:58:15.573 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T05:58:15.562+0000 7f2a6c693380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:58:15.573 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T05:58:15.574 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2021-09-17T05:58:15.576 INFO:tasks.rgw_multisite_tests:test s3 notification filter on master ... ok 2021-09-17T05:58:15.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_object_timing ... ok 2021-09-17T05:58:15.577 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-09-17T05:58:15.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T05:58:15.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T05:58:15.579 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T05:58:15.630 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T05:58:15.631 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T06:03:15.635 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:03:15.630+0000 7f1503517380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:08:15.633 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:08:15.630+0000 7f1503517380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:13:15.634 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:13:15.631+0000 7f1503517380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:18:15.634 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:18:15.632+0000 7f1503517380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:23:15.635 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:23:15.632+0000 7f1503517380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:28:15.635 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:28:15.632+0000 7f1503517380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:33:15.636 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:33:15.632+0000 7f1503517380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:38:15.636 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:38:15.634+0000 7f1503517380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:43:15.636 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:43:15.634+0000 7f1503517380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:48:15.637 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:48:15.634+0000 7f1503517380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:48:15.669 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T06:48:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T06:48:15.694 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2021-09-17T06:48:15.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T06:48:15.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T06:48:15.697 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T06:48:15.748 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T06:48:15.749 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T06:53:15.752 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:53:15.748+0000 7fe30c112380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:58:15.753 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T06:58:15.750+0000 7fe30c112380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:03:15.754 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:03:15.750+0000 7fe30c112380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:08:15.754 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:08:15.750+0000 7fe30c112380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:13:15.755 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:13:15.752+0000 7fe30c112380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:18:15.755 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:18:15.752+0000 7fe30c112380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:23:15.758 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:23:15.753+0000 7fe30c112380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:28:15.757 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:28:15.753+0000 7fe30c112380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:33:15.758 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:33:15.753+0000 7fe30c112380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:38:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T07:38:15.762 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:38:15.755+0000 7fe30c112380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:38:15.762 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T07:38:15.764 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2021-09-17T07:38:15.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T07:38:15.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T07:38:15.766 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T07:38:15.822 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T07:38:15.823 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T07:43:15.821 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:43:15.817+0000 7f66ff657380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:48:15.821 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:48:15.819+0000 7f66ff657380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:53:15.822 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:53:15.819+0000 7f66ff657380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:58:15.823 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T07:58:15.819+0000 7f66ff657380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:03:15.823 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:03:15.820+0000 7f66ff657380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:08:15.823 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:08:15.820+0000 7f66ff657380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:13:15.823 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:13:15.820+0000 7f66ff657380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:18:15.824 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:18:15.821+0000 7f66ff657380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:23:15.824 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:23:15.822+0000 7f66ff657380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:28:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T08:28:15.828 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:28:15.822+0000 7f66ff657380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:28:15.829 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T08:28:15.830 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2021-09-17T08:28:15.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T08:28:15.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T08:28:15.833 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T08:28:15.884 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T08:28:15.885 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T08:33:15.887 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:33:15.885+0000 7f4aa7972380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:38:15.888 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:38:15.885+0000 7f4aa7972380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:43:15.888 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:43:15.886+0000 7f4aa7972380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:48:15.889 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:48:15.885+0000 7f4aa7972380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:53:15.886 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:53:15.886+0000 7f4aa7972380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T08:58:15.884 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T08:58:15.888+0000 7f4aa7972380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:03:15.882 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:03:15.888+0000 7f4aa7972380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:08:15.882 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:08:15.889+0000 7f4aa7972380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:13:15.881 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:13:15.890+0000 7f4aa7972380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:18:15.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T09:18:15.885 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:18:15.890+0000 7f4aa7972380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:18:15.886 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T09:18:15.887 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2021-09-17T09:18:15.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T09:18:15.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T09:18:15.892 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T09:18:15.946 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T09:18:15.946 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T09:23:15.946 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:23:15.957+0000 7feaea5cb380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:28:15.946 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:28:15.958+0000 7feaea5cb380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:33:15.946 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:33:15.958+0000 7feaea5cb380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:38:15.946 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:38:15.958+0000 7feaea5cb380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:43:15.945 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:43:15.959+0000 7feaea5cb380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:48:15.945 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:48:15.959+0000 7feaea5cb380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:53:15.946 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:53:15.960+0000 7feaea5cb380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T09:58:15.945 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T09:58:15.961+0000 7feaea5cb380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:03:15.945 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:03:15.961+0000 7feaea5cb380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:08:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T10:08:15.948 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:08:15.961+0000 7feaea5cb380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:08:15.948 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T10:08:15.949 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... FAIL 2021-09-17T10:08:15.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:15.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:15.952 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:08:16.002 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:16.002 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:16.004 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:13:16.021+0000 7f96c5277380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:18:16.004 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:18:16.021+0000 7f96c5277380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:23:16.003 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:23:16.022+0000 7f96c5277380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:28:16.003 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:28:16.023+0000 7f96c5277380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:33:16.003 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:33:16.023+0000 7f96c5277380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:38:16.003 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:38:16.023+0000 7f96c5277380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:43:16.003 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:43:16.023+0000 7f96c5277380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:48:16.002 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:48:16.024+0000 7f96c5277380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:53:16.003 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:53:16.025+0000 7f96c5277380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:58:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T10:58:16.006 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T10:58:16.026+0000 7f96c5277380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T10:58:16.006 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T10:58:16.008 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2021-09-17T10:58:16.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:58:16.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:58:16.011 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:58:16.062 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:58:16.063 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T11:03:16.064 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:03:16.089+0000 7f56d781a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:08:16.073 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:08:16.089+0000 7f56d781a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:13:16.084 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:13:16.090+0000 7f56d781a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:18:16.091 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:18:16.091+0000 7f56d781a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:23:16.097 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:23:16.091+0000 7f56d781a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:28:16.101 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:28:16.092+0000 7f56d781a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:33:16.104 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:33:16.093+0000 7f56d781a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:38:16.106 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:38:16.093+0000 7f56d781a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:43:16.104 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:43:16.094+0000 7f56d781a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:48:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T11:48:16.126 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:48:16.095+0000 7f56d781a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:48:16.126 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T11:48:16.128 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... FAIL 2021-09-17T11:48:16.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T11:48:16.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T11:48:16.130 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T11:48:16.184 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T11:48:16.184 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T11:53:16.185 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:53:16.180+0000 7f39b2263380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:58:16.188 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T11:58:16.180+0000 7f39b2263380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:03:16.186 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:03:16.181+0000 7f39b2263380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:08:16.187 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:08:16.181+0000 7f39b2263380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:13:16.188 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:13:16.182+0000 7f39b2263380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:18:16.187 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:18:16.181+0000 7f39b2263380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:23:16.188 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:23:16.183+0000 7f39b2263380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:28:16.187 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:28:16.183+0000 7f39b2263380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:33:16.187 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:33:16.184+0000 7f39b2263380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:38:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T12:38:16.192 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:38:16.184+0000 7f39b2263380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:38:16.192 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T12:38:16.194 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... FAIL 2021-09-17T12:38:16.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T12:38:16.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T12:38:16.197 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T12:38:16.248 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T12:38:16.248 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T12:43:16.251 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:43:16.248+0000 7f600312f380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:48:16.252 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:48:16.248+0000 7f600312f380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:53:16.253 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:53:16.248+0000 7f600312f380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:58:16.253 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T12:58:16.249+0000 7f600312f380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:03:16.254 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T13:03:16.251+0000 7f600312f380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:08:16.254 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T13:08:16.252+0000 7f600312f380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:13:16.256 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T13:13:16.252+0000 7f600312f380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:18:16.256 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T13:18:16.253+0000 7f600312f380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:23:16.256 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T13:23:16.253+0000 7f600312f380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:28:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T13:28:16.260 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T13:28:16.253+0000 7f600312f380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:28:16.260 INFO:teuthology.orchestra.run.smithi013.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T13:28:16.262 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... FAIL 2021-09-17T13:28:16.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T13:28:16.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T13:28:16.265 DEBUG:teuthology.orchestra.run.smithi013:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T13:28:16.319 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setuser ceph since I am not root 2021-09-17T13:28:16.319 INFO:teuthology.orchestra.run.smithi013.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T13:33:16.322 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T13:33:16.318+0000 7fb07bcd9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:38:16.322 INFO:teuthology.orchestra.run.smithi013.stderr:2021-09-17T13:38:16.320+0000 7fb07bcd9380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:38:34.811 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-09-17T13:38:34.813 DEBUG:teuthology.task.console_log:Killing console logger for smithi013 2021-09-17T13:38:34.814 DEBUG:teuthology.task.console_log:Killing console logger for smithi045 2021-09-17T13:38:34.815 DEBUG:teuthology.exit:Finished running handlers